2023-11-16T00:48:37.736 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T00:48:37.736 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/orig.config.yaml 2023-11-16T00:48:37.806 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T00:48:37.900 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611 branch: wip-yuri3-testing-2023-11-14-1227-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7459611' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri3-testing-2023-11-14-1227-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-11-14-1227-pacific sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 7739 sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-11-14-1227-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac targets: smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyAg+yJz4Z2rYuRwd2DCKUgHJEtkIGtylF3x6Lt3xLtYxAsgDhPScARCgpz0UnjTxtTKtdDpRcIYSfSVmN5cDE= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQFNFq4x0TNBsuEXvW3zbKTAuxecTw3dBAgWgBx8ef4Klg+XKt5bO+aFQ7FJZ07K4DQ8kvQtf8GOMVo0EjJgBM= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-15_21:10:46 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-16T00:48:37.900 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa; will attempt to use it 2023-11-16T00:48:37.901 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks 2023-11-16T00:48:37.901 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T00:48:37.903 INFO:teuthology.task.internal:Checking packages... 2023-11-16T00:48:37.936 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac' 2023-11-16T00:48:37.936 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T00:48:37.936 INFO:teuthology.packaging:ref: None 2023-11-16T00:48:37.937 INFO:teuthology.packaging:tag: None 2023-11-16T00:48:37.937 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-16T00:48:37.937 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T00:48:37.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-16T00:48:38.081 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-171.ged3c8e92 2023-11-16T00:48:38.081 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T00:48:38.088 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T00:48:38.089 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T00:48:38.105 INFO:teuthology.task.internal:Saving configuration 2023-11-16T00:48:38.128 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T00:48:38.134 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T00:48:38.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 00:43:39.261038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyAg+yJz4Z2rYuRwd2DCKUgHJEtkIGtylF3x6Lt3xLtYxAsgDhPScARCgpz0UnjTxtTKtdDpRcIYSfSVmN5cDE='} 2023-11-16T00:48:38.185 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 00:43:39.258931', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQFNFq4x0TNBsuEXvW3zbKTAuxecTw3dBAgWgBx8ef4Klg+XKt5bO+aFQ7FJZ07K4DQ8kvQtf8GOMVo0EjJgBM='} 2023-11-16T00:48:38.185 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T00:48:38.191 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-16T00:48:38.191 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-16T00:48:38.192 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T00:48:38.292 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f99b375df70>, signals=[15]) 2023-11-16T00:48:38.293 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T00:48:38.308 INFO:teuthology.task.internal:Opening connections... 2023-11-16T00:48:38.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2023-11-16T00:48:38.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:48:38.390 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2023-11-16T00:48:38.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:48:38.459 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T00:48:38.466 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2023-11-16T00:48:38.488 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2023-11-16T00:48:38.488 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8" 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8" 2023-11-16T00:48:38.550 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T00:48:38.551 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-16T00:48:38.552 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-11-16T00:48:38.577 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2023-11-16T00:48:38.599 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2023-11-16T00:48:38.600 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2023-11-16T00:48:38.661 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-16T00:48:38.662 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2023-11-16T00:48:38.662 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T00:48:38.662 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T00:48:38.662 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-16T00:48:38.662 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2023-11-16T00:48:38.686 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T00:48:38.696 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T00:48:38.702 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T00:48:38.702 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2023-11-16T00:48:38.705 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2023-11-16T00:48:38.724 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T00:48:38.731 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T00:48:38.731 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2023-11-16T00:48:38.766 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2023-11-16T00:48:38.795 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T00:48:38.849 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T00:48:38.866 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-16T00:48:38.867 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-16T00:48:38.867 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T00:48:38.868 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T00:48:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T00:48:38.887 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2023-11-16T00:48:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T00:48:38.889 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-11-16T00:48:38.948 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-489.el8.x86_64 2023-11-16T00:48:38.948 INFO:teuthology.task.kernel:Running kernel on smithi134: 4.18.0-489.el8.x86_64 2023-11-16T00:48:38.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2023-11-16T00:48:38.950 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-489.el8.x86_64 2023-11-16T00:48:38.951 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.18.0-489.el8.x86_64 2023-11-16T00:48:38.951 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2023-11-16T00:48:46.308 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 7.0 MB/s | 34 MB 00:04 2023-11-16T00:48:50.750 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - AppStream 3.7 MB/s | 34 MB 00:09 2023-11-16T00:48:54.816 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-16T00:49:03.000 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2023-11-16T00:49:04.669 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 6.9 kB 00:00 2023-11-16T00:49:04.670 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-16T00:49:05.212 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-11-16T00:49:09.035 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 258 kB/s | 24 kB 00:00 2023-11-16T00:49:12.576 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-16T00:49:12.577 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-16T00:49:12.806 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - BaseOS 3.3 MB/s | 53 MB 00:15 2023-11-16T00:49:12.843 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T00:49:12.844 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout:Install 3 Packages 2023-11-16T00:49:12.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:12.846 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 M 2023-11-16T00:49:12.846 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 96 M 2023-11-16T00:49:12.846 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T00:49:14.861 INFO:teuthology.orchestra.run.smithi134.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 5.1 MB/s | 10 MB 00:01 2023-11-16T00:49:15.636 INFO:teuthology.orchestra.run.smithi134.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-11-16T00:49:19.053 INFO:teuthology.orchestra.run.smithi134.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 5.8 MB/s | 36 MB 00:06 2023-11-16T00:49:19.053 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T00:49:19.053 INFO:teuthology.orchestra.run.smithi134.stdout:Total 14 MB/s | 89 MB 00:06 2023-11-16T00:49:19.731 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T00:49:20.160 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T00:49:20.160 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T00:49:20.697 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-11-16T00:49:21.147 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2023-11-16T00:49:21.392 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T00:49:21.411 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T00:49:21.451 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-11-16T00:49:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T00:49:24.738 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:49:28.353 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:49:28.713 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:49:32.141 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:49:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:49:54.193 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:49:54.218 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:49:54.570 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:49:54.570 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:49:54.570 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:49:55.344 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:55.345 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T00:49:55.491 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T00:49:55.708 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T00:49:55.709 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T00:49:55.709 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-16T00:49:55.709 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T00:49:55.709 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T00:49:55.709 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-16T00:49:55.709 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T00:49:55.709 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T00:49:55.919 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T00:49:55.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T00:49:55.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2023-11-16T00:49:55.953 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.02 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 129.el8 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Group : System Environment/Base 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 9265586 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-16T00:49:55.954 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Relocations : (not relocatable) 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : CentOS BuildSystem 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2023-11-16T00:49:55.955 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T00:49:55.956 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T00:49:55.956 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T00:49:55.957 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 4.18.0-521.el8.x86_64 2023-11-16T00:49:55.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T00:49:57.544 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2023-11-16T00:49:59.777 INFO:teuthology.orchestra.run.smithi134.stderr:done 2023-11-16T00:49:59.778 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2023-11-16T00:49:59.810 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.SSeuK0DQdH 2023-11-16T00:49:59.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SSeuK0DQdH 2023-11-16T00:49:59.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.SSeuK0DQdH 2023-11-16T00:50:00.047 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.SSeuK0DQdH is 6KB 2023-11-16T00:50:00.097 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.SSeuK0DQdH 2023-11-16T00:50:00.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T00:50:00.197 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-16T00:50:00.197 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-16T00:50:00.197 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-16T00:50:00.197 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-16T00:50:00.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-16T00:50:00.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2023-11-16T00:50:30.359 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-16T00:50:30.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2023-11-16T00:50:30.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:51:30.363 DEBUG:teuthology.orchestra.remote:timed out 2023-11-16T00:51:34.615 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 123 kB/s | 16 MB 02:13 2023-11-16T00:51:39.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2023-11-16T00:51:39.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:51:39.509 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 277 kB/s | 24 kB 00:00 2023-11-16T00:51:39.859 DEBUG:teuthology.orchestra.run.smithi134:> true 2023-11-16T00:51:40.178 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2023-11-16T00:51:40.178 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-16T00:51:40.180 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2023-11-16T00:51:40.202 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-521.el8.x86_64 2023-11-16T00:51:40.224 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-16T00:51:40.225 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T00:51:40.225 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-16T00:51:41.226 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-16T00:51:41.226 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T00:51:41.557 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2023-11-16T00:51:41.574 DEBUG:teuthology.parallel:result is None 2023-11-16T00:51:42.157 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-16T00:51:42.157 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-16T00:51:42.430 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T00:51:42.431 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T00:51:42.432 INFO:teuthology.orchestra.run.smithi133.stdout:Install 3 Packages 2023-11-16T00:51:42.432 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:51:42.432 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 89 M 2023-11-16T00:51:42.432 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 96 M 2023-11-16T00:51:42.432 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T00:51:43.847 INFO:teuthology.orchestra.run.smithi133.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.2 MB/s | 10 MB 00:01 2023-11-16T00:51:46.306 INFO:teuthology.orchestra.run.smithi133.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 9.8 MB/s | 36 MB 00:03 2023-11-16T00:51:48.788 INFO:teuthology.orchestra.run.smithi133.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 7.0 MB/s | 43 MB 00:06 2023-11-16T00:51:48.789 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T00:51:48.789 INFO:teuthology.orchestra.run.smithi133.stdout:Total 14 MB/s | 89 MB 00:06 2023-11-16T00:51:49.464 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T00:51:49.892 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T00:51:49.892 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T00:51:51.124 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T00:51:51.159 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T00:51:53.927 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T00:51:54.493 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:51:58.088 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:51:58.476 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:52:01.903 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:52:02.163 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:52:24.099 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:52:24.121 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:52:24.485 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:52:24.485 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-16T00:52:24.485 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-16T00:52:25.336 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:52:25.337 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T00:52:25.600 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T00:52:25.814 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T00:52:25.814 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T00:52:25.814 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-16T00:52:25.814 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-16T00:52:25.814 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-16T00:52:25.815 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-16T00:52:25.815 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2023-11-16T00:52:25.815 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-11-16T00:52:26.025 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-16T00:52:26.025 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-16T00:52:26.025 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.02 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 129.el8 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Group : System Environment/Base 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 9265586 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-16T00:52:26.059 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Relocations : (not relocatable) 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : CentOS BuildSystem 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T00:52:26.060 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2023-11-16T00:52:26.061 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T00:52:26.061 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T00:52:26.061 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T00:52:26.062 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 4.18.0-521.el8.x86_64 2023-11-16T00:52:26.062 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T00:52:27.595 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2023-11-16T00:52:29.927 INFO:teuthology.orchestra.run.smithi133.stderr:done 2023-11-16T00:52:29.929 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-11-16T00:52:29.963 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.PSNBWFzdK6 2023-11-16T00:52:29.963 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PSNBWFzdK6 2023-11-16T00:52:30.039 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.PSNBWFzdK6 2023-11-16T00:52:30.182 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.PSNBWFzdK6 is 6KB 2023-11-16T00:52:30.230 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.PSNBWFzdK6 2023-11-16T00:52:30.247 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T00:52:30.312 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-16T00:52:30.313 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-16T00:52:30.313 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-16T00:52:30.313 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-16T00:52:30.314 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-16T00:52:30.464 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2023-11-16T00:53:00.467 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-16T00:53:00.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-11-16T00:53:00.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:54:00.470 DEBUG:teuthology.orchestra.remote:timed out 2023-11-16T00:54:18.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-11-16T00:54:18.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T00:54:18.429 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-11-16T00:54:18.696 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-11-16T00:54:18.696 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-16T00:54:18.696 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-11-16T00:54:18.717 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-521.el8.x86_64 2023-11-16T00:54:18.717 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-16T00:54:18.717 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T00:54:18.718 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-16T00:54:19.718 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-16T00:54:19.718 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-16T00:54:19.803 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2023-11-16T00:54:19.816 DEBUG:teuthology.parallel:result is None 2023-11-16T00:54:19.816 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T00:54:19.834 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T00:54:19.835 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T00:54:19.840 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T00:54:19.869 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T00:54:19.897 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T00:54:19.909 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T00:54:19.909 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T00:54:19.913 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T00:54:19.976 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T00:54:19.985 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T00:54:19.985 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T00:54:19.990 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T00:54:20.033 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T00:54:20.034 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T00:54:20.050 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T00:54:20.051 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T00:54:20.051 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T00:54:20.058 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T00:54:20.058 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T00:54:20.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T00:54:20.094 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T00:54:20.106 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T00:54:20.107 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T00:54:20.133 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T00:54:20.153 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T00:54:20.240 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T00:54:20.286 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T00:54:20.370 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T00:54:20.419 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T00:54:20.419 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T00:54:20.486 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T00:54:20.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T00:54:20.592 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T00:54:20.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T00:54:20.727 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T00:54:20.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T00:54:20.797 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2023-11-16T00:54:20.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2023-11-16T00:54:20.854 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T00:54:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T00:54:21.291 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T00:54:21.299 INFO:teuthology.task.internal:Starting timer... 2023-11-16T00:54:21.300 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T00:54:21.319 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T00:54:21.331 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T00:54:21.331 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2023-11-16T00:54:21.443 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2023-11-16T00:54:21.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2023-11-16T00:54:21.533 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2023-11-16T00:54:21.535 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T00:54:21.535 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2023-11-16T00:54:21.558 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2023-11-16T00:54:21.559 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2023-11-16T00:54:21.619 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2023-11-16T00:54:21.620 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2023-11-16T00:54:21.620 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T00:54:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T00:54:21.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T00:54:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T00:54:21.679 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T00:54:21.680 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2023-11-16T00:54:21.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2023-11-16T00:54:21.751 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T00:54:21.761 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-16T00:54:21.776 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-16T00:54:21.983 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T00:54:22.029 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T00:54:22.031 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi133.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T01:02:49.885 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi133.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2023-11-16T01:02:49.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-11-16T01:02:49.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:02:49.962 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-11-16T01:02:49.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-11-16T01:02:49.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2023-11-16T01:02:49.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T01:02:50.058 DEBUG:teuthology.orchestra.run.smithi134:> true 2023-11-16T01:02:50.124 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2023-11-16T01:02:50.124 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T01:02:50.136 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T01:02:50.136 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T01:02:50.136 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:02:50.139 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T01:02:50.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:02:50.168 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T01:02:50.182 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T01:02:50.195 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T01:02:50.208 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T01:02:50.265 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2023-11-16T01:02:50.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2023-11-16T01:02:50.276 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2023-11-16T01:02:50.285 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2023-11-16T01:02:50.290 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T01:02:50.300 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T01:02:50.304 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T01:02:50.313 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T01:02:50.373 INFO:teuthology.orchestra.run.smithi133.stderr:bash: ntpq: command not found 2023-11-16T01:02:50.376 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T01:02:50.377 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-11-16T01:02:50.377 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.377 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.377 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.377 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.385 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:02:50.386 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T01:02:50.401 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T01:02:50.402 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi133.front.sepia.ceph.com 2023-11-16T01:02:50.402 DEBUG:teuthology.orchestra.run.smithi133:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T01:02:50.403 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2023-11-16T01:02:50.403 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T01:02:51.293 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:41 ago on Thu 16 Nov 2023 01:01:10 AM UTC. 2023-11-16T01:02:51.306 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:37 ago on Thu 16 Nov 2023 01:01:14 AM UTC. 2023-11-16T01:02:53.376 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout:Resetting modules: 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools 2023-11-16T01:02:53.377 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:02:53.378 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:02:53.378 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:53.378 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:02:53.430 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout:Resetting modules: 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:53.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:02:53.630 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:02:53.685 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:02:54.578 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:44 ago on Thu 16 Nov 2023 01:01:10 AM UTC. 2023-11-16T01:02:54.633 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:40 ago on Thu 16 Nov 2023 01:01:14 AM UTC. 2023-11-16T01:02:56.669 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:02:56.738 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:02:56.846 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:56.847 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:02:56.847 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:56.847 INFO:teuthology.orchestra.run.smithi133.stdout:Installing group/module packages: 2023-11-16T01:02:56.847 INFO:teuthology.orchestra.run.smithi133.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-16T01:02:56.847 INFO:teuthology.orchestra.run.smithi133.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-16T01:02:56.848 INFO:teuthology.orchestra.run.smithi133.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout:Installing module profiles: 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools/common 2023-11-16T01:02:56.849 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout:Enabling module streams: 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools rhel8 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2023-11-16T01:02:56.850 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:02:56.851 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 27 M 2023-11-16T01:02:56.851 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 89 M 2023-11-16T01:02:56.851 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:Installing group/module packages: 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-16T01:02:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2023-11-16T01:02:56.918 INFO:teuthology.orchestra.run.smithi134.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout:Installing module profiles: 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools/common 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling module streams: 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools rhel8 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2023-11-16T01:02:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:02:56.920 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 27 M 2023-11-16T01:02:56.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 89 M 2023-11-16T01:02:56.920 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:02:57.280 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.3 MB/s | 750 kB 00:00 2023-11-16T01:02:57.282 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.9 MB/s | 750 kB 00:00 2023-11-16T01:02:57.314 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.8 MB/s | 1.0 MB 00:00 2023-11-16T01:02:57.324 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.4 MB/s | 1.0 MB 00:00 2023-11-16T01:02:57.397 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 323 kB/s | 38 kB 00:00 2023-11-16T01:02:57.416 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 283 kB/s | 38 kB 00:00 2023-11-16T01:02:57.491 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.4 MB/s | 241 kB 00:00 2023-11-16T01:02:57.537 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.1 MB/s | 241 kB 00:00 2023-11-16T01:02:57.683 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 927 kB/s | 178 kB 00:00 2023-11-16T01:02:57.715 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.0 MB/s | 178 kB 00:00 2023-11-16T01:02:57.783 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2023-11-16T01:02:57.806 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 271 kB/s | 25 kB 00:00 2023-11-16T01:02:57.984 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 469 kB/s | 94 kB 00:00 2023-11-16T01:02:58.015 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 450 kB/s | 94 kB 00:00 2023-11-16T01:02:58.101 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 5.7 MB/s | 3.9 MB 00:00 2023-11-16T01:02:58.232 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.5 MB/s | 9.6 MB 00:01 2023-11-16T01:02:58.316 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 4.3 MB/s | 3.9 MB 00:00 2023-11-16T01:02:58.371 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.2 MB/s | 9.6 MB 00:01 2023-11-16T01:02:58.424 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 487 kB/s | 53 kB 00:00 2023-11-16T01:02:58.485 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 451 kB/s | 53 kB 00:00 2023-11-16T01:02:58.541 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 350 kB/s | 41 kB 00:00 2023-11-16T01:02:58.551 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 5.6 MB/s | 2.5 MB 00:00 2023-11-16T01:02:58.643 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 258 kB/s | 41 kB 00:00 2023-11-16T01:02:58.833 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 4.2 MB/s | 2.5 MB 00:00 2023-11-16T01:02:59.336 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 6.0 MB/s | 8.1 MB 00:01 2023-11-16T01:02:59.336 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:02:59.336 INFO:teuthology.orchestra.run.smithi133.stdout:Total 11 MB/s | 27 MB 00:02 2023-11-16T01:02:59.484 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 5.5 MB/s | 8.1 MB 00:01 2023-11-16T01:02:59.484 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:02:59.484 INFO:teuthology.orchestra.run.smithi134.stdout:Total 10 MB/s | 27 MB 00:02 2023-11-16T01:02:59.603 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:02:59.624 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:02:59.624 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:02:59.753 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:02:59.774 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:02:59.775 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:02:59.889 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:02:59.892 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:03:00.039 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:03:00.042 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:03:00.586 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:03:00.759 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:03:00.802 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-16T01:03:00.980 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-16T01:03:00.991 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-16T01:03:01.154 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-16T01:03:01.355 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-16T01:03:01.374 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T01:03:01.589 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T01:03:01.903 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-16T01:03:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-16T01:03:03.129 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T01:03:03.216 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T01:03:03.366 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-16T01:03:03.478 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-16T01:03:03.586 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-16T01:03:03.682 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-16T01:03:03.849 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-16T01:03:03.960 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-16T01:03:04.233 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-16T01:03:04.347 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-16T01:03:05.138 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T01:03:05.243 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T01:03:05.381 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-16T01:03:05.482 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-16T01:03:05.561 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T01:03:05.671 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T01:03:05.994 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T01:03:05.994 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-16T01:03:05.994 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-16T01:03:05.994 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-16T01:03:05.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T01:03:05.996 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-16T01:03:06.091 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-16T01:03:06.091 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-16T01:03:06.091 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-16T01:03:06.091 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-16T01:03:06.092 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-16T01:03:06.093 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-16T01:03:06.093 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-16T01:03:06.915 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T01:03:06.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:03:06.917 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:03:06.933 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T01:03:06.934 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:03:06.935 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:03:07.199 DEBUG:teuthology.parallel:result is None 2023-11-16T01:03:07.217 DEBUG:teuthology.parallel:result is None 2023-11-16T01:03:07.217 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T01:03:07.235 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T01:03:07.236 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi133.front.sepia.ceph.com 2023-11-16T01:03:07.236 DEBUG:teuthology.orchestra.run.smithi133:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T01:03:07.236 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2023-11-16T01:03:07.236 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T01:03:07.409 DEBUG:teuthology.parallel:result is None 2023-11-16T01:03:07.417 DEBUG:teuthology.parallel:result is None 2023-11-16T01:03:07.417 INFO:teuthology.run_tasks:Running task install... 2023-11-16T01:03:07.427 DEBUG:teuthology.task.install:project ceph 2023-11-16T01:03:07.427 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}} 2023-11-16T01:03:07.427 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-16T01:03:07.428 INFO:teuthology.task.install:Using flavor: default 2023-11-16T01:03:07.433 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T01:03:07.434 INFO:teuthology.task.install:extra packages: [] 2023-11-16T01:03:07.434 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-16T01:03:07.434 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:03:07.434 INFO:teuthology.packaging:ref: None 2023-11-16T01:03:07.434 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:03:07.434 INFO:teuthology.packaging:branch: None 2023-11-16T01:03:07.434 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:03:07.655 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:03:07.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:03:07.657 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-16T01:03:07.658 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:03:07.658 INFO:teuthology.packaging:ref: None 2023-11-16T01:03:07.658 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:03:07.658 INFO:teuthology.packaging:branch: None 2023-11-16T01:03:07.658 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:03:07.890 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:03:07.891 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:03:08.111 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-16T01:03:08.111 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-16T01:03:08.126 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-16T01:03:08.126 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-16T01:03:08.158 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T01:03:08.158 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:03:08.158 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T01:03:08.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T01:03:08.174 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:03:08.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T01:03:08.192 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-16T01:03:08.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:03:08.193 INFO:teuthology.packaging:ref: None 2023-11-16T01:03:08.193 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:03:08.193 INFO:teuthology.packaging:branch: None 2023-11-16T01:03:08.193 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:03:08.193 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T01:03:08.208 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-16T01:03:08.208 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:03:08.208 INFO:teuthology.packaging:ref: None 2023-11-16T01:03:08.208 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:03:08.208 INFO:teuthology.packaging:branch: None 2023-11-16T01:03:08.208 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:03:08.209 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T01:03:08.262 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T01:03:08.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T01:03:08.341 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T01:03:08.361 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T01:03:08.409 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2023-11-16T01:03:08.410 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2023-11-16T01:03:08.426 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2023-11-16T01:03:08.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2023-11-16T01:03:08.681 INFO:teuthology.orchestra.run.smithi133.stdout:76 files removed 2023-11-16T01:03:08.695 INFO:teuthology.orchestra.run.smithi134.stdout:76 files removed 2023-11-16T01:03:08.709 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2023-11-16T01:03:08.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2023-11-16T01:03:09.203 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 846 kB/s | 74 kB 00:00 2023-11-16T01:03:09.216 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 932 kB/s | 74 kB 00:00 2023-11-16T01:03:09.322 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 190 kB/s | 15 kB 00:00 2023-11-16T01:03:09.349 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 159 kB/s | 15 kB 00:00 2023-11-16T01:03:09.404 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 32 kB/s | 1.7 kB 00:00 2023-11-16T01:03:09.459 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2023-11-16T01:03:12.234 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-16T01:03:12.296 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-16T01:03:22.365 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 53 MB 00:03 2023-11-16T01:03:24.075 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 9.5 MB/s | 53 MB 00:05 2023-11-16T01:03:30.239 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras 58 kB/s | 18 kB 00:00 2023-11-16T01:03:30.823 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-11-16T01:03:32.125 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-16T01:03:32.760 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-11-16T01:03:34.381 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - PowerTools 1.9 MB/s | 6.6 MB 00:03 2023-11-16T01:03:35.086 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - PowerTools 3.1 MB/s | 6.6 MB 00:02 2023-11-16T01:03:35.821 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-16T01:03:36.520 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-16T01:03:37.796 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-16T01:03:41.696 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 463 kB/s | 24 kB 00:00 2023-11-16T01:03:46.203 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-16T01:03:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-16T01:03:46.207 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-16T01:03:46.208 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:Install 25 Packages 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 59 M 2023-11-16T01:03:46.209 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:03:46.598 INFO:teuthology.orchestra.run.smithi134.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 44 MB/s | 6.0 MB 00:00 2023-11-16T01:03:46.614 INFO:teuthology.orchestra.run.smithi134.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-11-16T01:03:46.648 INFO:teuthology.orchestra.run.smithi134.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-16T01:03:46.682 INFO:teuthology.orchestra.run.smithi134.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-16T01:03:46.816 INFO:teuthology.orchestra.run.smithi134.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.1 MB 00:00 2023-11-16T01:03:46.841 INFO:teuthology.orchestra.run.smithi134.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-16T01:03:46.866 INFO:teuthology.orchestra.run.smithi134.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-16T01:03:46.891 INFO:teuthology.orchestra.run.smithi134.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-16T01:03:46.917 INFO:teuthology.orchestra.run.smithi134.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2023-11-16T01:03:46.950 INFO:teuthology.orchestra.run.smithi134.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-16T01:03:46.976 INFO:teuthology.orchestra.run.smithi134.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2023-11-16T01:03:47.152 INFO:teuthology.orchestra.run.smithi134.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-16T01:03:47.452 INFO:teuthology.orchestra.run.smithi134.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-11-16T01:03:47.476 INFO:teuthology.orchestra.run.smithi134.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 290 kB/s | 145 kB 00:00 2023-11-16T01:03:47.518 INFO:teuthology.orchestra.run.smithi134.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 965 kB/s | 354 kB 00:00 2023-11-16T01:03:47.610 INFO:teuthology.orchestra.run.smithi134.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 328 kB/s | 44 kB 00:00 2023-11-16T01:03:47.652 INFO:teuthology.orchestra.run.smithi134.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2023-11-16T01:03:47.853 INFO:teuthology.orchestra.run.smithi134.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 126 kB/s | 42 kB 00:00 2023-11-16T01:03:47.947 INFO:teuthology.orchestra.run.smithi134.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 141 kB/s | 47 kB 00:00 2023-11-16T01:03:47.987 INFO:teuthology.orchestra.run.smithi134.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 240 kB/s | 80 kB 00:00 2023-11-16T01:03:48.012 INFO:teuthology.orchestra.run.smithi134.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 733 kB/s | 116 kB 00:00 2023-11-16T01:03:48.070 INFO:teuthology.orchestra.run.smithi134.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-11-16T01:03:48.129 INFO:teuthology.orchestra.run.smithi134.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-11-16T01:03:48.163 INFO:teuthology.orchestra.run.smithi134.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-11-16T01:03:48.188 INFO:teuthology.orchestra.run.smithi134.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 900 kB/s | 181 kB 00:00 2023-11-16T01:03:48.263 INFO:teuthology.orchestra.run.smithi134.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.0 MB 00:00 2023-11-16T01:03:48.322 INFO:teuthology.orchestra.run.smithi134.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-11-16T01:03:48.324 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:03:48.324 INFO:teuthology.orchestra.run.smithi134.stdout:Total 28 MB/s | 59 MB 00:02 2023-11-16T01:03:48.389 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:03:48.436 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:03:48.436 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:03:48.943 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:03:48.946 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:03:50.153 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:03:50.375 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-16T01:03:50.621 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-16T01:03:50.821 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-16T01:03:51.232 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-16T01:03:51.454 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:03:51.547 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:03:51.809 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-16T01:03:51.965 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:03:52.053 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:03:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-16T01:03:52.445 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-16T01:03:52.848 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-16T01:03:53.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:03:53.086 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:03:53.242 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-16T01:03:53.445 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-16T01:03:53.498 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-16T01:03:53.684 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-16T01:03:53.846 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-16T01:03:54.054 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-16T01:03:54.107 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-16T01:03:54.295 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-16T01:03:54.707 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-16T01:03:54.929 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-16T01:03:55.028 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-16T01:03:55.234 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-16T01:03:55.446 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-16T01:03:55.620 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-16T01:03:55.692 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-16T01:03:55.903 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-16T01:03:56.071 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-16T01:03:56.268 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-16T01:03:56.512 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-16T01:03:59.028 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:03:59.270 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:03:59.963 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:04:00.287 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-16T01:04:00.396 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-16T01:04:00.598 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:09.634 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 485 kB/s | 16 MB 00:33 2023-11-16T01:04:13.448 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 499 kB/s | 24 kB 00:00 2023-11-16T01:04:18.052 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:04:18.055 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:18.055 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:04:18.055 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:18.055 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:04:18.055 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-16T01:04:18.056 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-16T01:04:18.057 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout:Install 25 Packages 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2023-11-16T01:04:18.058 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:18.059 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 59 M 2023-11-16T01:04:18.059 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:04:18.782 INFO:teuthology.orchestra.run.smithi133.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2023-11-16T01:04:18.807 INFO:teuthology.orchestra.run.smithi133.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-16T01:04:18.841 INFO:teuthology.orchestra.run.smithi133.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-16T01:04:18.874 INFO:teuthology.orchestra.run.smithi133.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-16T01:04:18.966 INFO:teuthology.orchestra.run.smithi133.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2023-11-16T01:04:18.991 INFO:teuthology.orchestra.run.smithi133.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-16T01:04:19.017 INFO:teuthology.orchestra.run.smithi133.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-16T01:04:19.066 INFO:teuthology.orchestra.run.smithi133.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 213 kB 00:00 2023-11-16T01:04:19.101 INFO:teuthology.orchestra.run.smithi133.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 10 MB/s | 387 kB 00:00 2023-11-16T01:04:19.126 INFO:teuthology.orchestra.run.smithi133.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-11-16T01:04:19.151 INFO:teuthology.orchestra.run.smithi133.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /run 2023-11-16T01:04:19.247 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2023-11-16T01:04:19.248 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2023-11-16T01:04:19.248 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2023-11-16T01:04:19.248 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2023-11-16T01:04:19.248 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2023-11-16T01:04:19.248 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:19.277 INFO:teuthology.orchestra.run.smithi133.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2023-11-16T01:04:19.577 INFO:teuthology.orchestra.run.smithi133.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2023-11-16T01:04:19.602 INFO:teuthology.orchestra.run.smithi133.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 322 kB/s | 145 kB 00:00 2023-11-16T01:04:19.836 INFO:teuthology.orchestra.run.smithi133.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 634 kB/s | 354 kB 00:00 2023-11-16T01:04:19.853 INFO:teuthology.orchestra.run.smithi133.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 175 kB/s | 44 kB 00:00 2023-11-16T01:04:20.045 INFO:teuthology.orchestra.run.smithi133.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 556 kB/s | 259 kB 00:00 2023-11-16T01:04:20.345 INFO:teuthology.orchestra.run.smithi133.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 83 kB/s | 42 kB 00:00 2023-11-16T01:04:20.362 INFO:teuthology.orchestra.run.smithi133.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 92 kB/s | 47 kB 00:00 2023-11-16T01:04:20.470 INFO:teuthology.orchestra.run.smithi133.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 188 kB/s | 80 kB 00:00 2023-11-16T01:04:20.617 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-16T01:04:20.705 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-16T01:04:20.706 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-16T01:04:20.944 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-16T01:04:20.944 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:20.971 INFO:teuthology.orchestra.run.smithi133.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 191 kB/s | 116 kB 00:00 2023-11-16T01:04:21.121 INFO:teuthology.orchestra.run.smithi133.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 404 kB/s | 306 kB 00:00 2023-11-16T01:04:21.171 INFO:teuthology.orchestra.run.smithi133.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 293 kB/s | 59 kB 00:00 2023-11-16T01:04:21.197 INFO:teuthology.orchestra.run.smithi133.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 252 kB/s | 181 kB 00:00 2023-11-16T01:04:21.280 INFO:teuthology.orchestra.run.smithi133.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-11-16T01:04:21.306 INFO:teuthology.orchestra.run.smithi133.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 407 kB/s | 75 kB 00:00 2023-11-16T01:04:21.356 INFO:teuthology.orchestra.run.smithi133.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 25 MB/s | 4.0 MB 00:00 2023-11-16T01:04:21.360 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:04:21.360 INFO:teuthology.orchestra.run.smithi133.stdout:Total 18 MB/s | 59 MB 00:03 2023-11-16T01:04:21.426 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:04:21.474 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:04:21.474 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-16T01:04:21.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-16T01:04:21.676 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-16T01:04:21.677 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-16T01:04:21.678 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-16T01:04:21.817 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:04:21.821 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:04:22.913 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb-1.22-1.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-16T01:04:22.914 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:22.915 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:22.916 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:04:23.000 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:04:23.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2023-11-16T01:04:23.188 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-16T01:04:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-16T01:04:23.567 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-16T01:04:24.013 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-16T01:04:24.050 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:43 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:04:24.208 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:04:24.290 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:04:24.565 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-16T01:04:24.708 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:04:24.758 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:04:24.981 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-16T01:04:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:04:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:33.720 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:04:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:04:33.721 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-16T01:04:33.722 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 229 M 2023-11-16T01:04:33.723 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 169 kB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 263 kB/s | 203 kB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 267 kB/s | 187 kB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 249 kB/s | 71 kB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 646 kB/s | 323 kB 00:00 2023-11-16T01:04:33.724 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Total 36 MB/s | 53 MB 00:01 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-16T01:04:33.725 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T01:04:33.756 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T01:04:33.757 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-16T01:04:33.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-16T01:04:33.781 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-16T01:04:33.781 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-16T01:04:33.781 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:04:33.781 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-16T01:04:33.782 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-16T01:04:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:34.022 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-16T01:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-16T01:04:34.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-16T01:04:35.356 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:35.357 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:04:35.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2023-11-16T01:04:36.436 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:04:38.086 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:04:38.089 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:38.089 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:04:38.089 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:38.089 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T01:04:38.090 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-16T01:04:38.091 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:38.092 INFO:teuthology.orchestra.run.smithi134.stdout:Install 32 Packages 2023-11-16T01:04:38.093 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:38.093 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 M 2023-11-16T01:04:38.093 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 111 M 2023-11-16T01:04:38.093 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:04:38.316 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2023-11-16T01:04:38.349 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-16.2.4-0.el8.x86_64.rpm 45 kB/s | 6.3 kB 00:00 2023-11-16T01:04:38.391 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2023-11-16T01:04:38.416 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-16T01:04:38.500 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 27 MB/s | 5.0 MB 00:00 2023-11-16T01:04:38.758 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2023-11-16T01:04:38.875 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 342 kB/s | 157 kB 00:00 2023-11-16T01:04:38.942 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 88 kB/s | 39 kB 00:00 2023-11-16T01:04:38.967 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 496 kB/s | 103 kB 00:00 2023-11-16T01:04:39.043 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 341 kB/s | 57 kB 00:00 2023-11-16T01:04:39.293 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2023-11-16T01:04:39.560 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 420 kB/s | 249 kB 00:00 2023-11-16T01:04:39.618 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-11-16T01:04:39.694 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 272 kB/s | 109 kB 00:00 2023-11-16T01:04:39.777 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 201 kB/s | 44 kB 00:00 2023-11-16T01:04:39.878 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 716 kB/s | 185 kB 00:00 2023-11-16T01:04:39.953 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2023-11-16T01:04:39.978 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 608 kB/s | 173 kB 00:00 2023-11-16T01:04:40.004 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-11-16T01:04:40.029 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 232 kB/s | 18 kB 00:00 2023-11-16T01:04:40.054 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 643 kB/s | 49 kB 00:00 2023-11-16T01:04:40.079 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2023-11-16T01:04:40.143 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 175 kB/s | 16 kB 00:00 2023-11-16T01:04:40.171 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 204 kB/s | 19 kB 00:00 2023-11-16T01:04:40.197 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-16T01:04:40.222 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 305 kB/s | 24 kB 00:00 2023-11-16T01:04:40.248 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2023-11-16T01:04:40.273 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2023-11-16T01:04:40.323 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-11-16T01:04:40.348 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-11-16T01:04:40.374 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 852 kB/s | 85 kB 00:00 2023-11-16T01:04:40.393 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 344 kB/s | 23 kB 00:00 2023-11-16T01:04:40.394 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:04:40.394 INFO:teuthology.orchestra.run.smithi134.stdout:Total 13 MB/s | 31 MB 00:02 2023-11-16T01:04:40.475 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:04:40.514 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:04:40.514 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:04:40.815 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:04:40.818 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:04:41.993 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:04:42.209 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-16T01:04:42.454 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-16T01:04:42.728 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-16T01:04:42.977 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-16T01:04:43.158 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-16T01:04:43.380 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-16T01:04:43.582 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-16T01:04:43.776 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-16T01:04:44.009 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-16T01:04:44.247 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-16T01:04:44.461 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-16T01:04:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-16T01:04:45.105 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-16T01:04:45.382 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-16T01:04:45.578 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-16T01:04:45.878 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-16T01:04:46.038 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T01:04:46.346 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-16T01:04:46.609 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-16T01:04:46.806 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-16T01:04:47.010 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-16T01:04:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-16T01:04:47.544 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-16T01:04:47.780 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-16T01:04:48.128 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-16T01:04:48.476 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-16T01:04:48.892 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-16T01:04:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-16T01:04:50.653 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-16T01:04:50.895 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-16T01:04:51.328 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /run 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2023-11-16T01:04:51.503 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:51.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-16T01:04:51.776 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-16T01:04:51.978 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-16T01:04:51.996 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-16T01:04:52.128 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-16T01:04:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-16T01:04:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-16T01:04:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-16T01:04:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-16T01:04:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-16T01:04:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-16T01:04:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-16T01:04:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-16T01:04:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-16T01:04:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-16T01:04:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-16T01:04:52.873 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-16T01:04:52.946 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-16T01:04:52.947 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-16T01:04:53.189 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-16T01:04:53.189 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:53.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-16T01:04:53.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T01:04:53.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T01:04:53.810 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T01:04:53.810 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:53.810 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:04:53.878 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:53.878 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-16T01:04:53.878 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-16T01:04:53.878 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-16T01:04:53.879 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-16T01:04:53.880 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-16T01:04:53.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2023-11-16T01:04:54.813 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:13 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:04:55.090 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-16T01:04:55.090 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:55.091 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2023-11-16T01:04:55.091 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.091 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:04:55.092 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.092 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.092 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.093 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.093 INFO:teuthology.orchestra.run.smithi133.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-16T01:04:55.094 INFO:teuthology.orchestra.run.smithi133.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-16T01:04:55.094 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T01:04:55.094 INFO:teuthology.orchestra.run.smithi133.stdout: leveldb-1.22-1.el8.x86_64 2023-11-16T01:04:55.095 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.096 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T01:04:55.096 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-16T01:04:55.098 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-16T01:04:55.098 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.099 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T01:04:55.099 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.100 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T01:04:55.100 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T01:04:55.101 INFO:teuthology.orchestra.run.smithi133.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-16T01:04:55.102 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.102 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.102 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.103 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T01:04:55.103 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.103 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.104 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-16T01:04:55.104 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:55.104 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:04:55.349 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2023-11-16T01:04:56.204 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:43 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:04:56.264 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:04:56.484 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:04:56.485 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:04:56.485 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:04:56.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2023-11-16T01:04:57.443 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:16 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:04:57.876 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-16T01:04:57.877 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:04:57.878 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 53 M 2023-11-16T01:04:57.879 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 229 M 2023-11-16T01:04:57.879 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:04:57.987 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2023-11-16T01:04:58.120 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2023-11-16T01:04:58.162 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-11-16T01:04:58.747 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2023-11-16T01:04:58.772 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 116 kB/s | 71 kB 00:00 2023-11-16T01:04:58.830 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 456 kB/s | 323 kB 00:00 2023-11-16T01:04:58.832 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:04:58.832 INFO:teuthology.orchestra.run.smithi133.stdout:Total 55 MB/s | 53 MB 00:00 2023-11-16T01:04:58.875 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:04:58.896 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:04:58.896 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:04:59.118 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-16T01:04:59.120 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 71 k 2023-11-16T01:04:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 290 k 2023-11-16T01:04:59.122 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:04:59.161 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2023-11-16T01:04:59.162 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:04:59.162 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.7 MB/s | 71 kB 00:00 2023-11-16T01:04:59.163 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:04:59.173 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:04:59.173 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:04:59.173 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:04:59.174 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:04:59.206 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:04:59.213 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:04:59.218 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:04:59.715 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:04:59.717 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:04:59.963 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-16T01:05:00.154 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-16T01:05:00.360 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T01:05:00.365 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:00.443 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-16T01:05:00.597 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:00.659 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-16T01:05:00.802 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-16T01:05:01.262 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:01.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:01.263 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:01.263 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-16T01:05:01.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:01.263 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:01.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2023-11-16T01:05:02.396 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:21 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:04.045 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:05:04.046 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:04.047 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:05:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:04.047 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 163 k 2023-11-16T01:05:04.047 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 587 k 2023-11-16T01:05:04.047 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:05:04.125 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.1 MB/s | 163 kB 00:00 2023-11-16T01:05:04.125 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:04.126 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 163 kB 00:00 2023-11-16T01:05:04.126 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:05:04.136 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:05:04.136 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:05:04.184 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:05:04.187 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:05:04.300 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:05:04.509 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:04.911 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:05.574 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:05.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:05.574 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:05.574 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:05.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:05.575 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:05.662 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-16T01:05:05.891 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-16T01:05:06.217 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-16T01:05:06.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-16T01:05:06.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-16T01:05:06.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-16T01:05:06.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-16T01:05:06.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:06.921 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:07.156 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2023-11-16T01:05:07.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2023-11-16T01:05:08.012 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:55 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:08.312 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:27 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:09.685 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-16T01:05:09.689 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T01:05:09.690 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T01:05:09.691 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T01:05:09.692 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T01:05:09.693 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T01:05:09.694 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T01:05:09.694 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-16T01:05:09.694 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T01:05:09.694 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:09.694 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:05:09.695 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:09.695 INFO:teuthology.orchestra.run.smithi133.stdout:Install 32 Packages 2023-11-16T01:05:09.696 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:09.697 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 31 M 2023-11-16T01:05:09.697 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 111 M 2023-11-16T01:05:09.697 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:05:09.775 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:05:09.881 INFO:teuthology.orchestra.run.smithi133.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 129 kB/s | 6.3 kB 00:00 2023-11-16T01:05:09.932 INFO:teuthology.orchestra.run.smithi133.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-16T01:05:09.990 INFO:teuthology.orchestra.run.smithi133.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-16T01:05:09.991 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:09.992 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:05:09.992 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:10.016 INFO:teuthology.orchestra.run.smithi133.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-11-16T01:05:10.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2023-11-16T01:05:10.141 INFO:teuthology.orchestra.run.smithi133.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 19 MB/s | 5.0 MB 00:00 2023-11-16T01:05:10.375 INFO:teuthology.orchestra.run.smithi133.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 41 MB/s | 18 MB 00:00 2023-11-16T01:05:10.400 INFO:teuthology.orchestra.run.smithi133.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2023-11-16T01:05:10.425 INFO:teuthology.orchestra.run.smithi133.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 384 kB/s | 157 kB 00:00 2023-11-16T01:05:10.542 INFO:teuthology.orchestra.run.smithi133.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 402 kB/s | 57 kB 00:00 2023-11-16T01:05:10.567 INFO:teuthology.orchestra.run.smithi133.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 538 kB/s | 103 kB 00:00 2023-11-16T01:05:10.709 INFO:teuthology.orchestra.run.smithi133.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2023-11-16T01:05:10.948 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:11.060 INFO:teuthology.orchestra.run.smithi133.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 481 kB/s | 249 kB 00:00 2023-11-16T01:05:11.102 INFO:teuthology.orchestra.run.smithi133.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.2 MB/s | 641 kB 00:00 2023-11-16T01:05:11.128 INFO:teuthology.orchestra.run.smithi133.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 261 kB/s | 109 kB 00:00 2023-11-16T01:05:11.569 INFO:teuthology.orchestra.run.smithi133.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2023-11-16T01:05:11.702 INFO:teuthology.orchestra.run.smithi133.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 308 kB/s | 185 kB 00:00 2023-11-16T01:05:11.753 INFO:teuthology.orchestra.run.smithi133.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 276 kB/s | 173 kB 00:00 2023-11-16T01:05:11.803 INFO:teuthology.orchestra.run.smithi133.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-11-16T01:05:11.853 INFO:teuthology.orchestra.run.smithi133.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-11-16T01:05:11.914 INFO:teuthology.orchestra.run.smithi133.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 450 kB/s | 49 kB 00:00 2023-11-16T01:05:11.954 INFO:teuthology.orchestra.run.smithi133.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 585 kB/s | 59 kB 00:00 2023-11-16T01:05:12.062 INFO:teuthology.orchestra.run.smithi133.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-11-16T01:05:12.179 INFO:teuthology.orchestra.run.smithi133.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2023-11-16T01:05:12.204 INFO:teuthology.orchestra.run.smithi133.stdout:(24/32): python3-cherrypy-18.4.0-1.el8.noarch.r 605 kB/s | 384 kB 00:00 2023-11-16T01:05:12.281 INFO:teuthology.orchestra.run.smithi133.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 239 kB/s | 24 kB 00:00 2023-11-16T01:05:12.313 INFO:teuthology.orchestra.run.smithi133.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2023-11-16T01:05:12.339 INFO:teuthology.orchestra.run.smithi133.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 663 kB/s | 283 kB 00:00 2023-11-16T01:05:12.380 INFO:teuthology.orchestra.run.smithi133.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-11-16T01:05:12.581 INFO:teuthology.orchestra.run.smithi133.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 427 kB/s | 85 kB 00:00 2023-11-16T01:05:12.617 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:12.618 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:12.619 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 15 M 2023-11-16T01:05:12.620 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:05:12.689 INFO:teuthology.orchestra.run.smithi133.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 717 kB/s | 251 kB 00:00 2023-11-16T01:05:12.714 INFO:teuthology.orchestra.run.smithi133.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 637 kB/s | 255 kB 00:00 2023-11-16T01:05:12.740 INFO:teuthology.orchestra.run.smithi133.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 145 kB/s | 23 kB 00:00 2023-11-16T01:05:12.744 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:12.745 INFO:teuthology.orchestra.run.smithi133.stdout:Total 10 MB/s | 31 MB 00:03 2023-11-16T01:05:12.826 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:05:12.865 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:05:12.865 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:05:12.869 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 394 kB/s | 24 kB 00:00 2023-11-16T01:05:12.894 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 103 kB/s | 8.9 kB 00:00 2023-11-16T01:05:12.986 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-11-16T01:05:13.185 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:05:13.188 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:05:13.262 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 109 kB/s | 43 kB 00:00 2023-11-16T01:05:14.321 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:05:14.521 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-16T01:05:14.757 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-16T01:05:14.999 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-16T01:05:15.247 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-16T01:05:15.429 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-16T01:05:15.635 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-16T01:05:15.811 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-16T01:05:16.013 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-16T01:05:16.214 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-16T01:05:16.468 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-16T01:05:16.703 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-16T01:05:16.971 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-16T01:05:17.303 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-16T01:05:17.551 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-16T01:05:17.740 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-16T01:05:17.981 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-16T01:05:18.167 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T01:05:18.441 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-16T01:05:18.663 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-16T01:05:18.892 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-16T01:05:19.077 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-16T01:05:19.347 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-16T01:05:19.561 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-16T01:05:19.820 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-16T01:05:20.070 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 4.7 kB/s | 34 kB 00:07 2023-11-16T01:05:20.145 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-16T01:05:20.237 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 27 kB/s | 196 kB 00:07 2023-11-16T01:05:20.240 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:20.240 INFO:teuthology.orchestra.run.smithi134.stdout:Total 396 kB/s | 3.0 MB 00:07 2023-11-16T01:05:20.255 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:05:20.262 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:05:20.262 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:05:20.441 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:05:20.444 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:05:20.445 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-16T01:05:20.754 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:05:20.856 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-16T01:05:20.988 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-16T01:05:21.101 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-16T01:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-16T01:05:21.475 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-16T01:05:21.639 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-16T01:05:22.619 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-16T01:05:22.630 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-16T01:05:22.851 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-16T01:05:23.284 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-16T01:05:23.525 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:23.551 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-16T01:05:23.774 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-16T01:05:24.087 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-16T01:05:24.087 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-16T01:05:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-16T01:05:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-16T01:05:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-16T01:05:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-16T01:05:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-16T01:05:24.136 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-16T01:05:24.533 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-16T01:05:24.533 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-16T01:05:24.534 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-16T01:05:24.534 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-16T01:05:24.535 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-16T01:05:24.536 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-16T01:05:24.537 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-16T01:05:24.538 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-16T01:05:24.772 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T01:05:24.773 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T01:05:24.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:24.800 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:24.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-16T01:05:25.765 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:25.768 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-16T01:05:25.768 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:25.768 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:05:25.768 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-16T01:05:25.769 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T01:05:25.770 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:25.771 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:26.026 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2023-11-16T01:05:26.887 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:13 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:27.437 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================================ 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================================ 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-16T01:05:27.438 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================================ 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout:Install 8 Packages 2023-11-16T01:05:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:27.440 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2023-11-16T01:05:27.440 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 210 M 2023-11-16T01:05:27.440 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:05:27.859 INFO:teuthology.orchestra.run.smithi134.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 64 MB/s | 8.6 MB 00:00 2023-11-16T01:05:28.348 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:05:28.563 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:28.564 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:05:28.564 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:28.666 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2023-11-16T01:05:29.334 INFO:teuthology.orchestra.run.smithi134.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 2.9 MB/s | 4.7 MB 00:01 2023-11-16T01:05:29.417 INFO:teuthology.orchestra.run.smithi134.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.9 MB/s | 4.9 MB 00:01 2023-11-16T01:05:29.512 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:16 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:29.618 INFO:teuthology.orchestra.run.smithi134.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 794 kB/s | 225 kB 00:00 2023-11-16T01:05:30.202 INFO:teuthology.orchestra.run.smithi134.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.6 MB/s | 3.7 MB 00:02 2023-11-16T01:05:30.235 INFO:teuthology.orchestra.run.smithi134.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2023-11-16T01:05:30.669 INFO:teuthology.orchestra.run.smithi134.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 11 MB/s | 14 MB 00:01 2023-11-16T01:05:30.752 INFO:teuthology.orchestra.run.smithi134.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 314 kB/s | 172 kB 00:00 2023-11-16T01:05:30.753 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:30.753 INFO:teuthology.orchestra.run.smithi134.stdout:Total 11 MB/s | 37 MB 00:03 2023-11-16T01:05:30.983 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:05:31.001 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:05:31.001 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:05:31.240 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:31.240 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:31.240 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:05:31.240 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:31.240 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 71 k 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 290 k 2023-11-16T01:05:31.241 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:05:31.286 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-11-16T01:05:31.286 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:31.287 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-11-16T01:05:31.287 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:05:31.293 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:05:31.293 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:05:31.295 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:05:31.298 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:05:31.331 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:05:31.335 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:05:31.345 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:05:31.833 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:05:32.122 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T01:05:32.260 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T01:05:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T01:05:32.736 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:32.970 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:33.033 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T01:05:33.214 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:33.685 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:33.844 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:05:33.899 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2023-11-16T01:05:34.038 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T01:05:34.731 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T01:05:34.762 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:21 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:35.090 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T01:05:35.365 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T01:05:36.425 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:36.425 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:36.425 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2023-11-16T01:05:36.425 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:36.425 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:05:36.426 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:36.427 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 163 k 2023-11-16T01:05:36.427 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 587 k 2023-11-16T01:05:36.427 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:05:36.474 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-11-16T01:05:36.475 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:36.475 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-11-16T01:05:36.475 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:05:36.485 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:05:36.486 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:05:36.532 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:05:36.539 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:05:36.642 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:05:36.846 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:37.129 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T01:05:37.265 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:37.910 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:38.157 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2023-11-16T01:05:38.452 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-16T01:05:38.665 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-16T01:05:38.987 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-16T01:05:38.987 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-16T01:05:38.987 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-16T01:05:38.987 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:05:38.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-16T01:05:38.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T01:05:38.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-16T01:05:38.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-16T01:05:39.066 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:26 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:39.679 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-16T01:05:39.679 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:39.679 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:39.679 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-16T01:05:39.679 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-16T01:05:39.680 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-16T01:05:39.680 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-16T01:05:39.680 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-16T01:05:39.680 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T01:05:39.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T01:05:39.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-16T01:05:39.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:39.682 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:39.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2023-11-16T01:05:40.524 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:05:40.655 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:59 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:40.741 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:05:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:40.850 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2023-11-16T01:05:41.701 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:28 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:42.386 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:42.387 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:42.387 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:05:42.387 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-16T01:05:42.388 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2023-11-16T01:05:42.389 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:42.390 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.9 M 2023-11-16T01:05:42.390 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2023-11-16T01:05:42.390 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:05:42.999 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 691 kB/s | 27 kB 00:00 2023-11-16T01:05:43.074 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 175 kB/s | 20 kB 00:00 2023-11-16T01:05:43.091 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 203 kB/s | 27 kB 00:00 2023-11-16T01:05:43.175 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 721 kB/s | 126 kB 00:00 2023-11-16T01:05:43.225 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 731 kB/s | 110 kB 00:00 2023-11-16T01:05:43.351 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-16T01:05:43.352 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2023-11-16T01:05:43.353 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:43.354 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.0 M 2023-11-16T01:05:43.354 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 15 M 2023-11-16T01:05:43.355 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:05:43.392 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 198 kB/s | 33 kB 00:00 2023-11-16T01:05:43.476 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 196 kB/s | 16 kB 00:00 2023-11-16T01:05:43.493 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2023-11-16T01:05:43.593 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 694 kB/s | 81 kB 00:00 2023-11-16T01:05:43.660 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 965 kB/s | 64 kB 00:00 2023-11-16T01:05:43.694 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 257 kB/s | 155 kB 00:00 2023-11-16T01:05:43.752 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 448 kB/s | 24 kB 00:00 2023-11-16T01:05:43.758 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 627 kB/s | 61 kB 00:00 2023-11-16T01:05:43.777 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 114 kB/s | 8.9 kB 00:00 2023-11-16T01:05:43.836 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2023-11-16T01:05:43.919 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-11-16T01:05:43.936 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.6 MB/s | 1.2 MB 00:00 2023-11-16T01:05:43.936 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:43.936 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-11-16T01:05:43.995 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:05:44.011 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:05:44.011 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:05:44.028 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-11-16T01:05:44.095 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 124 kB/s | 43 kB 00:00 2023-11-16T01:05:44.096 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:44.097 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.0 MB/s | 3.0 MB 00:00 2023-11-16T01:05:44.097 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:05:44.101 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:05:44.106 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:05:44.116 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:05:44.116 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:05:44.260 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:05:44.263 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:05:44.561 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:05:44.623 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:05:44.766 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-16T01:05:44.889 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-16T01:05:45.095 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-16T01:05:45.122 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-16T01:05:45.329 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-16T01:05:45.388 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-16T01:05:45.463 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-16T01:05:45.659 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-16T01:05:45.863 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-16T01:05:46.125 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-16T01:05:46.433 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-16T01:05:46.434 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-16T01:05:46.717 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-16T01:05:46.738 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:47.018 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:47.018 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-16T01:05:47.018 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-16T01:05:47.018 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-16T01:05:47.019 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-16T01:05:47.019 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-16T01:05:47.695 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T01:05:47.696 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T01:05:47.696 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T01:05:47.696 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:47.696 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:05:47.798 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-16T01:05:47.836 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-16T01:05:48.084 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-16T01:05:48.320 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-16T01:05:48.527 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-16T01:05:48.677 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-16T01:05:48.689 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:35 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-16T01:05:49.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-16T01:05:49.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-16T01:05:49.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-16T01:05:49.797 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T01:05:49.798 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:49.799 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:49.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2023-11-16T01:05:50.345 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:05:50.346 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================================ 2023-11-16T01:05:50.346 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:05:50.346 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================================ 2023-11-16T01:05:50.346 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================================ 2023-11-16T01:05:50.347 INFO:teuthology.orchestra.run.smithi133.stdout:Install 8 Packages 2023-11-16T01:05:50.348 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:05:50.348 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 37 M 2023-11-16T01:05:50.348 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 210 M 2023-11-16T01:05:50.348 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:05:50.759 INFO:teuthology.orchestra.run.smithi133.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 54 MB/s | 8.6 MB 00:00 2023-11-16T01:05:50.769 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:09 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:52.035 INFO:teuthology.orchestra.run.smithi133.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.3 MB/s | 4.7 MB 00:01 2023-11-16T01:05:52.245 INFO:teuthology.orchestra.run.smithi133.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.5 MB/s | 3.7 MB 00:01 2023-11-16T01:05:52.277 INFO:teuthology.orchestra.run.smithi133.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 931 kB/s | 225 kB 00:00 2023-11-16T01:05:52.460 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-16T01:05:52.461 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2023-11-16T01:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.8 M 2023-11-16T01:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2023-11-16T01:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:05:52.519 INFO:teuthology.orchestra.run.smithi133.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.5 MB/s | 4.9 MB 00:01 2023-11-16T01:05:52.809 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-11-16T01:05:53.003 INFO:teuthology.orchestra.run.smithi133.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 889 kB/s | 645 kB 00:00 2023-11-16T01:05:53.086 INFO:teuthology.orchestra.run.smithi133.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 304 kB/s | 172 kB 00:00 2023-11-16T01:05:53.093 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 75 kB/s | 21 kB 00:00 2023-11-16T01:05:53.334 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2023-11-16T01:05:53.401 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 733 kB/s | 49 kB 00:00 2023-11-16T01:05:53.427 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 802 kB/s | 538 kB 00:00 2023-11-16T01:05:54.110 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 3.5 MB/s | 4.8 MB 00:01 2023-11-16T01:05:54.112 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:54.112 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.5 MB/s | 5.8 MB 00:01 2023-11-16T01:05:54.164 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:05:54.173 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:05:54.173 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:05:54.276 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:05:54.279 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:05:54.386 INFO:teuthology.orchestra.run.smithi133.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 6.4 MB/s | 14 MB 00:02 2023-11-16T01:05:54.388 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:05:54.388 INFO:teuthology.orchestra.run.smithi133.stdout:Total 9.1 MB/s | 37 MB 00:04 2023-11-16T01:05:54.578 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:05:54.622 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:05:54.640 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:05:54.640 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:05:54.969 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:05:54.972 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:05:55.186 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-16T01:05:55.476 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-16T01:05:55.780 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:05:56.424 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-16T01:05:56.459 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T01:05:56.592 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-16T01:05:56.734 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T01:05:56.818 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-16T01:05:57.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-16T01:05:57.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:57.369 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-16T01:05:57.542 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:05:57.583 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:05:57.583 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-16T01:05:57.583 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-16T01:05:57.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-16T01:05:57.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-16T01:05:57.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-16T01:05:58.183 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:05:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:05:58.341 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T01:05:58.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2023-11-16T01:05:59.022 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-16T01:05:59.258 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:18 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:05:59.395 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T01:05:59.649 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-16T01:06:00.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 968 k 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.2 M 2023-11-16T01:06:00.919 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:01.196 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 9.2 MB/s | 885 kB 00:00 2023-11-16T01:06:01.416 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-16T01:06:01.454 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 236 kB/s | 83 kB 00:00 2023-11-16T01:06:01.455 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:01.455 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.8 MB/s | 968 kB 00:00 2023-11-16T01:06:01.465 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:01.486 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:01.486 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:01.572 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:01.575 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:01.743 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:02.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-16T01:06:02.242 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-16T01:06:02.642 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-16T01:06:02.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-16T01:06:02.830 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-16T01:06:03.048 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-16T01:06:03.321 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-16T01:06:03.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:03.322 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:03.322 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-16T01:06:03.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:03.322 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:03.357 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-16T01:06:03.357 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-16T01:06:03.358 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-16T01:06:03.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2023-11-16T01:06:04.070 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-16T01:06:04.070 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:04.070 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:04.071 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:04.176 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2023-11-16T01:06:04.426 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:23 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:05.032 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:52 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:06.074 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-16T01:06:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 131 k 2023-11-16T01:06:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 439 k 2023-11-16T01:06:06.077 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:06.418 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 383 kB/s | 131 kB 00:00 2023-11-16T01:06:06.419 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:06.419 INFO:teuthology.orchestra.run.smithi134.stdout:Total 382 kB/s | 131 kB 00:00 2023-11-16T01:06:06.419 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:06.424 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:06.425 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:06.474 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:06.477 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:06.587 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:06.707 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-16T01:06:06.709 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:06.710 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.9 M 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 24 M 2023-11-16T01:06:06.711 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:06:06.840 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:07.017 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 584 kB/s | 27 kB 00:00 2023-11-16T01:06:07.125 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 129 kB/s | 20 kB 00:00 2023-11-16T01:06:07.131 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:07.151 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 148 kB/s | 27 kB 00:00 2023-11-16T01:06:07.209 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 657 kB/s | 126 kB 00:00 2023-11-16T01:06:07.285 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 692 kB/s | 110 kB 00:00 2023-11-16T01:06:07.418 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 247 kB/s | 33 kB 00:00 2023-11-16T01:06:07.507 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 184 kB/s | 16 kB 00:00 2023-11-16T01:06:07.544 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 128 kB/s | 43 kB 00:00 2023-11-16T01:06:07.569 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 370 kB/s | 155 kB 00:00 2023-11-16T01:06:07.628 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 675 kB/s | 81 kB 00:00 2023-11-16T01:06:07.695 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 921 kB/s | 61 kB 00:00 2023-11-16T01:06:07.737 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 385 kB/s | 64 kB 00:00 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:07.807 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:07.862 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.7 MB/s | 1.2 MB 00:00 2023-11-16T01:06:07.862 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:07.862 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-11-16T01:06:07.922 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:06:07.937 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:06:07.938 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:06:08.024 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:06:08.027 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:06:08.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2023-11-16T01:06:08.569 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:06:08.826 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-16T01:06:08.947 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:27 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:09.026 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-16T01:06:09.284 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-16T01:06:09.488 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-16T01:06:09.747 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-16T01:06:09.987 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-16T01:06:10.298 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-16T01:06:10.403 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:10.588 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-16T01:06:10.618 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:10.619 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:10.619 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:10.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2023-11-16T01:06:11.601 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:30 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:11.677 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-16T01:06:11.938 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-16T01:06:12.162 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-16T01:06:12.355 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-16T01:06:12.564 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-16T01:06:12.874 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-16T01:06:12.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-16T01:06:12.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-16T01:06:13.260 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:13.260 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:13.260 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:06:13.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:13.262 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 24 k 2023-11-16T01:06:13.262 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 94 k 2023-11-16T01:06:13.262 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:13.314 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 468 kB/s | 24 kB 00:00 2023-11-16T01:06:13.314 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:13.314 INFO:teuthology.orchestra.run.smithi134.stdout:Total 457 kB/s | 24 kB 00:00 2023-11-16T01:06:13.315 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:13.318 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:13.318 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:13.320 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:13.323 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:13.377 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:13.605 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:13.619 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-16T01:06:13.619 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:13.619 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:13.619 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-16T01:06:13.619 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T01:06:13.620 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:13.621 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:13.747 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2023-11-16T01:06:13.853 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:14.513 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:14.603 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:01 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:14.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2023-11-16T01:06:15.493 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:34 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:16.291 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-16T01:06:16.292 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2023-11-16T01:06:16.293 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:16.294 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.8 M 2023-11-16T01:06:16.294 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 24 M 2023-11-16T01:06:16.294 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:06:16.610 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 89 kB 00:00 2023-11-16T01:06:16.902 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2023-11-16T01:06:16.927 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 67 kB/s | 21 kB 00:00 2023-11-16T01:06:16.999 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:17.214 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:17.215 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:17.215 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:17.238 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.0 MB/s | 355 kB 00:00 2023-11-16T01:06:17.288 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 135 kB/s | 49 kB 00:00 2023-11-16T01:06:17.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2023-11-16T01:06:17.547 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.8 MB/s | 4.8 MB 00:00 2023-11-16T01:06:17.550 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:17.550 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.6 MB/s | 5.8 MB 00:01 2023-11-16T01:06:17.601 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:06:17.611 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:06:17.611 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:06:17.710 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:06:17.714 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:06:18.033 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:06:18.193 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:37 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:18.346 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-16T01:06:18.659 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-16T01:06:19.599 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-16T01:06:19.615 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:19.833 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:19.834 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:19.834 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:19.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2023-11-16T01:06:19.940 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-16T01:06:20.158 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-16T01:06:20.387 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:06:20.665 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-16T01:06:20.666 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-16T01:06:20.666 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-16T01:06:20.666 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-16T01:06:20.666 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-16T01:06:20.667 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-16T01:06:20.794 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:39 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:21.373 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-16T01:06:21.373 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:21.373 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-16T01:06:21.373 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-16T01:06:21.374 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-16T01:06:21.374 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:21.500 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2023-11-16T01:06:22.248 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:22.367 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:09 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:22.461 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:22.462 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:22.462 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:22.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2023-11-16T01:06:23.424 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:42 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:24.021 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:24.021 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:24.021 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2023-11-16T01:06:24.022 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:24.023 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 968 k 2023-11-16T01:06:24.023 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 3.2 M 2023-11-16T01:06:24.023 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:06:24.274 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-16T01:06:24.732 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 162 kB/s | 83 kB 00:00 2023-11-16T01:06:24.733 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:24.733 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.3 MB/s | 968 kB 00:00 2023-11-16T01:06:24.744 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:06:24.765 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:06:24.765 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:06:24.853 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:06:24.856 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:06:24.863 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:25.029 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:06:25.077 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:25.078 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:25.078 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:25.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2023-11-16T01:06:25.296 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-16T01:06:25.479 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-16T01:06:25.867 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-16T01:06:25.867 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-16T01:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:45 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:26.519 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:26.756 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2023-11-16T01:06:27.481 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:27.630 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:14 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:27.698 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:27.699 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:27.699 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:27.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2023-11-16T01:06:28.688 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:47 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:29.302 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:06:29.303 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:29.304 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 131 k 2023-11-16T01:06:29.304 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 439 k 2023-11-16T01:06:29.304 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:06:29.344 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 131 kB 00:00 2023-11-16T01:06:29.345 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:29.345 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.1 MB/s | 131 kB 00:00 2023-11-16T01:06:29.345 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:06:29.350 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:06:29.351 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:06:29.401 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:06:29.404 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:06:29.520 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:06:29.744 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:30.073 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:30.131 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:30.346 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:30.347 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2023-11-16T01:06:30.347 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:30.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2023-11-16T01:06:30.770 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:30.770 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:30.770 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:30.770 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:30.771 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:30.771 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:30.986 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2023-11-16T01:06:31.334 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:50 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:31.854 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:18 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:33.021 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:06:33.023 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:33.023 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 k 2023-11-16T01:06:33.023 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 k 2023-11-16T01:06:33.023 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:33.299 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:33.308 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 316 kB/s | 89 kB 00:00 2023-11-16T01:06:33.308 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:33.309 INFO:teuthology.orchestra.run.smithi134.stdout:Total 314 kB/s | 89 kB 00:00 2023-11-16T01:06:33.309 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:33.319 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:33.319 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:33.369 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:33.372 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:33.487 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:33.513 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:33.515 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:33.515 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:33.703 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:33.976 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:34.173 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2023-11-16T01:06:34.674 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:34.675 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:34.675 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:34.675 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:34.675 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:34.675 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:34.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2023-11-16T01:06:35.046 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:22 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:35.743 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:54 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:36.714 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:36.714 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:36.714 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:06:36.715 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:36.716 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 24 k 2023-11-16T01:06:36.716 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 94 k 2023-11-16T01:06:36.716 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:06:36.758 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 575 kB/s | 24 kB 00:00 2023-11-16T01:06:36.761 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:Total 559 kB/s | 24 kB 00:00 2023-11-16T01:06:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:06:36.762 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:06:36.763 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:06:36.765 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:06:36.768 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:06:36.845 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:06:37.034 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:37.301 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:37.404 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:37.404 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:37.404 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:37.404 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:06:37.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:37.406 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.7 M 2023-11-16T01:06:37.406 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 18 M 2023-11-16T01:06:37.406 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:37.481 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2023-11-16T01:06:37.482 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:37.482 INFO:teuthology.orchestra.run.smithi134.stdout:Total 48 MB/s | 3.7 MB 00:00 2023-11-16T01:06:37.482 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:37.494 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:37.494 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:37.559 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:37.562 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:37.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:37.934 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:37.934 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:06:37.934 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:37.935 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:06:37.935 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:38.053 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:38.061 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2023-11-16T01:06:38.370 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:38.793 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:38.921 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:25 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:39.448 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:39.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2023-11-16T01:06:40.352 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:40.567 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:40.568 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:40.568 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:40.587 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:59 ago on Thu 16 Nov 2023 01:03:41 AM UTC. 2023-11-16T01:06:40.669 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2023-11-16T01:06:41.529 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:28 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:42.260 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2023-11-16T01:06:42.261 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:42.261 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2023-11-16T01:06:42.261 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:42.261 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2023-11-16T01:06:42.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:42.263 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 173 k 2023-11-16T01:06:42.263 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 649 k 2023-11-16T01:06:42.263 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2023-11-16T01:06:42.318 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 173 kB 00:00 2023-11-16T01:06:42.318 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:06:42.318 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.0 MB/s | 173 kB 00:00 2023-11-16T01:06:42.319 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2023-11-16T01:06:42.329 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2023-11-16T01:06:42.329 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2023-11-16T01:06:42.380 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2023-11-16T01:06:42.383 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2023-11-16T01:06:42.502 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2023-11-16T01:06:42.726 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:42.973 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:43.004 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:43.191 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:43.193 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:43.193 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:43.319 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2023-11-16T01:06:43.661 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:06:43.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:43.661 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2023-11-16T01:06:43.661 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-16T01:06:43.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:06:43.662 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2023-11-16T01:06:43.888 DEBUG:teuthology.parallel:result is None 2023-11-16T01:06:44.183 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:31 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:45.623 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:45.842 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:45.844 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:45.844 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:45.959 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2023-11-16T01:06:46.815 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:33 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:48.264 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:48.479 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:48.480 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:48.480 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:48.575 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2023-11-16T01:06:49.433 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:36 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:50.873 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:51.089 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:51.091 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:51.091 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:51.197 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2023-11-16T01:06:52.055 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:39 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:06:53.497 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-16T01:06:53.711 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:06:53.712 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-11-16T01:06:53.712 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:06:53.819 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2023-11-16T01:06:54.680 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:41 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:07:01.505 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:07:01.538 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:01.538 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:07:01.538 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:01.538 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:07:01.538 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 89 k 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 298 k 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 89 kB 00:00 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.3 MB/s | 89 kB 00:00 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:07:01.539 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:01.540 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:07:01.541 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2023-11-16T01:07:02.425 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:49 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:07:04.088 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:07:04.088 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:04.088 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:07:04.088 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.7 M 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 18 M 2023-11-16T01:07:04.089 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:07:04.174 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2023-11-16T01:07:04.174 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:07:04.175 INFO:teuthology.orchestra.run.smithi133.stdout:Total 43 MB/s | 3.7 MB 00:00 2023-11-16T01:07:04.175 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:07:04.187 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:07:04.187 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:07:04.251 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:07:04.254 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:07:04.753 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:07:04.963 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:05.383 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:06.033 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:07:06.221 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2023-11-16T01:07:07.086 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:02:54 ago on Thu 16 Nov 2023 01:04:13 AM UTC. 2023-11-16T01:07:08.727 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-11-16T01:07:08.730 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:08.730 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-11-16T01:07:08.730 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:08.731 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-11-16T01:07:08.731 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-16T01:07:08.731 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:08.731 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-11-16T01:07:08.731 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-11-16T01:07:08.732 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2023-11-16T01:07:08.732 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:08.732 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 173 k 2023-11-16T01:07:08.732 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 649 k 2023-11-16T01:07:08.732 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-11-16T01:07:08.769 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 173 kB 00:00 2023-11-16T01:07:08.769 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-11-16T01:07:08.770 INFO:teuthology.orchestra.run.smithi133.stdout:Total 4.1 MB/s | 173 kB 00:00 2023-11-16T01:07:08.770 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-11-16T01:07:08.780 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-11-16T01:07:08.780 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-11-16T01:07:08.831 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-11-16T01:07:08.834 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-11-16T01:07:08.969 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-11-16T01:07:09.136 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:09.436 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:10.120 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-16T01:07:10.120 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:10.120 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-11-16T01:07:10.120 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-16T01:07:10.120 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-11-16T01:07:10.121 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-11-16T01:07:10.330 DEBUG:teuthology.parallel:result is None 2023-11-16T01:07:10.331 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:07:10.331 INFO:teuthology.packaging:ref: None 2023-11-16T01:07:10.331 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:07:10.331 INFO:teuthology.packaging:branch: None 2023-11-16T01:07:10.331 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:07:10.534 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:07:10.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:07:10.678 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T01:07:10.702 INFO:teuthology.orchestra.run.smithi133.stdout:16.2.4-0.el8 2023-11-16T01:07:10.702 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-16T01:07:10.702 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-16T01:07:10.703 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-16T01:07:10.703 INFO:teuthology.packaging:ref: None 2023-11-16T01:07:10.703 INFO:teuthology.packaging:tag: v16.2.4 2023-11-16T01:07:10.703 INFO:teuthology.packaging:branch: None 2023-11-16T01:07:10.703 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-16T01:07:10.898 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:07:10.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-16T01:07:11.124 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T01:07:11.152 INFO:teuthology.orchestra.run.smithi134.stdout:16.2.4-0.el8 2023-11-16T01:07:11.152 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-16T01:07:11.152 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-16T01:07:11.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T01:07:11.153 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:07:11.153 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T01:07:11.186 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:07:11.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T01:07:11.225 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T01:07:11.225 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:07:11.225 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T01:07:11.258 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T01:07:11.327 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:07:11.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T01:07:11.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T01:07:11.430 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T01:07:11.430 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:07:11.430 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T01:07:11.459 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T01:07:11.528 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:07:11.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T01:07:11.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T01:07:11.632 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T01:07:11.632 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:07:11.632 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T01:07:11.663 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T01:07:11.733 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:07:11.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T01:07:11.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T01:07:11.836 INFO:teuthology.run_tasks:Running task print... 2023-11-16T01:07:11.849 INFO:teuthology.task.print:**** done install task... 2023-11-16T01:07:11.849 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T01:07:11.944 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-16T01:07:11.944 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-16T01:07:11.944 INFO:tasks.cephadm:Cluster fsid is 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:11.944 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T01:07:11.944 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-16T01:07:11.945 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi133': '172.21.15.133', 'mon.smithi134': '172.21.15.134'} 2023-11-16T01:07:11.945 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T01:07:11.945 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname $(hostname -s) 2023-11-16T01:07:11.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2023-11-16T01:07:12.009 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-16T01:07:12.010 DEBUG:teuthology.orchestra.run.smithi133:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.153 INFO:teuthology.orchestra.run.smithi133.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 16 01:07 /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.154 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.300 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 16 01:07 /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.300 DEBUG:teuthology.orchestra.run.smithi133:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.323 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T01:07:12.356 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph 2023-11-16T01:07:12.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2023-11-16T01:07:12.428 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /etc/ceph 2023-11-16T01:07:12.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2023-11-16T01:07:12.498 INFO:tasks.cephadm:Writing seed config... 2023-11-16T01:07:12.499 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T01:07:12.499 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T01:07:12.499 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T01:07:12.500 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-16T01:07:12.501 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T01:07:12.502 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T01:07:12.503 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T01:07:12.504 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:07:12.504 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T01:07:12.529 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7905ec40-841c-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T01:07:12.529 DEBUG:teuthology.orchestra.run.smithi133:mon.smithi133> sudo journalctl -f -n 0 -u ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service 2023-11-16T01:07:12.572 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T01:07:12.573 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.133 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T01:07:12.603 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:-- Logs begin at Thu 2023-11-16 00:53:54 UTC. -- 2023-11-16T01:07:12.850 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-16T01:07:13.712 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: 4.6.1 2023-11-16T01:07:13.723 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying podman|docker is present... 2023-11-16T01:07:13.723 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-16T01:07:13.746 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: 4.6.1 2023-11-16T01:07:13.756 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying lvm2 is present... 2023-11-16T01:07:13.756 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying time synchronization is in place... 2023-11-16T01:07:13.757 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled chrony.service 2023-11-16T01:07:13.763 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T01:07:13.764 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active chrony.service 2023-11-16T01:07:13.770 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:07:13.771 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-16T01:07:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: enabled 2023-11-16T01:07:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active chronyd.service 2023-11-16T01:07:13.782 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: active 2023-11-16T01:07:13.783 INFO:teuthology.orchestra.run.smithi133.stderr:Unit chronyd.service is enabled and running 2023-11-16T01:07:13.783 INFO:teuthology.orchestra.run.smithi133.stderr:Repeating the final host check... 2023-11-16T01:07:13.783 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-16T01:07:13.803 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: 4.6.1 2023-11-16T01:07:13.814 INFO:teuthology.orchestra.run.smithi133.stderr:podman|docker (/bin/podman) is present 2023-11-16T01:07:13.814 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl is present 2023-11-16T01:07:13.814 INFO:teuthology.orchestra.run.smithi133.stderr:lvcreate is present 2023-11-16T01:07:13.814 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled chrony.service 2023-11-16T01:07:13.820 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T01:07:13.821 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active chrony.service 2023-11-16T01:07:13.826 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:07:13.827 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-16T01:07:13.832 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: enabled 2023-11-16T01:07:13.833 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active chronyd.service 2023-11-16T01:07:13.838 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: active 2023-11-16T01:07:13.839 INFO:teuthology.orchestra.run.smithi133.stderr:Unit chronyd.service is enabled and running 2023-11-16T01:07:13.839 INFO:teuthology.orchestra.run.smithi133.stderr:Host looks OK 2023-11-16T01:07:13.839 INFO:teuthology.orchestra.run.smithi133.stderr:Cluster fsid: 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:13.839 INFO:teuthology.orchestra.run.smithi133.stderr:Acquiring lock 140568729164264 on /run/cephadm/7905ec40-841c-11ee-8dbe-212e2dc638e7.lock 2023-11-16T01:07:13.839 INFO:teuthology.orchestra.run.smithi133.stderr:Lock 140568729164264 acquired on /run/cephadm/7905ec40-841c-11ee-8dbe-212e2dc638e7.lock 2023-11-16T01:07:13.840 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying IP 172.21.15.133 port 3300 ... 2023-11-16T01:07:13.840 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying IP 172.21.15.133 port 6789 ... 2023-11-16T01:07:13.840 INFO:teuthology.orchestra.run.smithi133.stderr:Base mon IP is 172.21.15.133, final addrv is [v2:172.21.15.133:3300,v1:172.21.15.133:6789] 2023-11-16T01:07:13.840 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /sbin/ip route ls 2023-11-16T01:07:13.842 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.133 metric 100 2023-11-16T01:07:13.842 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.133 metric 100 2023-11-16T01:07:13.843 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /sbin/ip -6 route ls 2023-11-16T01:07:13.845 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T01:07:13.845 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-16T01:07:13.845 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2023-11-16T01:07:13.846 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /sbin/ip -6 addr ls 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:8087/64 scope link 2023-11-16T01:07:13.849 INFO:teuthology.orchestra.run.smithi133.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-16T01:07:13.850 INFO:teuthology.orchestra.run.smithi133.stderr:Mon IP 172.21.15.133 is in CIDR network 172.21.0.0/20 2023-11-16T01:07:13.850 INFO:teuthology.orchestra.run.smithi133.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T01:07:13.850 INFO:teuthology.orchestra.run.smithi133.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-16T01:07:13.850 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-16T01:07:13.965 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-16T01:07:20.441 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Getting image source signatures 2023-11-16T01:07:20.456 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-16T01:07:20.456 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-16T01:07:32.259 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-16T01:07:32.325 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: Writing manifest to image destination 2023-11-16T01:07:32.647 INFO:teuthology.orchestra.run.smithi133.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-16T01:07:32.659 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-16T01:07:33.193 INFO:teuthology.orchestra.run.smithi133.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-16T01:07:33.544 INFO:teuthology.orchestra.run.smithi133.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-16T01:07:33.544 INFO:teuthology.orchestra.run.smithi133.stderr:Extracting ceph user uid/gid from container image... 2023-11-16T01:07:33.544 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-16T01:07:33.868 INFO:teuthology.orchestra.run.smithi133.stderr:stat: 167 167 2023-11-16T01:07:34.184 INFO:teuthology.orchestra.run.smithi133.stderr:Creating initial keys... 2023-11-16T01:07:34.184 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-16T01:07:34.499 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-authtool: AQBWa1VluG+UHBAAQe1umqxWYUECJ6hPuc95Aw== 2023-11-16T01:07:34.844 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-16T01:07:35.107 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-authtool: AQBXa1Vl22fgBBAAksBeI7k7M56CXTBMOaaa2w== 2023-11-16T01:07:35.429 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-16T01:07:35.741 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-authtool: AQBXa1VlBCBHKxAAeJd9ydNExSaxGNTWQlR/uw== 2023-11-16T01:07:36.078 INFO:teuthology.orchestra.run.smithi133.stderr:Creating initial monmap... 2023-11-16T01:07:36.078 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmptmogqyew:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 --addv smithi133 [v2:172.21.15.133:3300,v1:172.21.15.133:6789] /tmp/monmap 2023-11-16T01:07:36.359 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T01:07:36.360 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:36.360 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T01:07:36.700 INFO:teuthology.orchestra.run.smithi133.stderr:monmaptool for smithi133 [v2:172.21.15.133:3300,v1:172.21.15.133:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T01:07:36.700 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/monmaptool: set fsid to 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:36.700 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T01:07:36.701 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:07:36.701 INFO:teuthology.orchestra.run.smithi133.stderr:Creating mon... 2023-11-16T01:07:36.701 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/mon.smithi133:/var/lib/ceph/mon/ceph-smithi133:z -v /tmp/ceph-tmpo99ftmoa:/tmp/keyring:z -v /tmp/ceph-tmptmogqyew:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi133 --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.946+0000 7f7829250700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.946+0000 7f7829250700 1 imported monmap: 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: last_changed 2023-11-16T01:07:36.340362+0000 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: created 2023-11-16T01:07:36.340362+0000 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-16T01:07:36.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.smithi133 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.946+0000 7f7829250700 0 /usr/bin/ceph-mon: set fsid to 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: RocksDB version: 6.8.1 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Compile date May 13 2021 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: DB SUMMARY 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi133/store.db dir, Total Num: 0, files: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi133/store.db: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.error_if_exists: 0 2023-11-16T01:07:36.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.create_if_missing: 1 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.paranoid_checks: 1 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.env: 0x564a5baf61c0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.fs: Posix File System 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.info_log: 0x564a5e1d8ec0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.statistics: (nil) 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.use_fsync: 0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_log_file_size: 0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-16T01:07:36.992 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.allow_fallocate: 1 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.use_direct_reads: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.db_log_dir: 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi133/store.db 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_subcompactions: 1 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_background_flushes: -1 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T01:07:36.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.write_buffer_manager: 0x564a5e1df530 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-16T01:07:36.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.unordered_write: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.row_cache: None 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.wal_filter: None 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.preserve_deletes: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.two_write_queues: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.atomic_flush: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T01:07:36.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.log_readahead_size: 0 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_background_jobs: 2 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_background_compactions: -1 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T01:07:36.996 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.max_open_files: -1 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Compression algorithms supported: 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kZSTD supported: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kXpressCompression supported: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kLZ4Compression supported: 1 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kBZip2Compression supported: 0 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kZlibCompression supported: 1 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: kSnappyCompression supported: 1 2023-11-16T01:07:36.997 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.947+0000 7f7829250700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000001 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.merge_operator: 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_filter: None 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564a5e0f0120) 2023-11-16T01:07:36.998 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_cache: 0x564a5e126d10 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-16T01:07:36.999 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-16T01:07:37.000 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression: NoCompression 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.num_levels: 7 2023-11-16T01:07:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T01:07:37.002 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T01:07:37.003 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T01:07:37.004 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.table_properties_collectors: 2023-11-16T01:07:37.005 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.inplace_update_support: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.bloom_locality: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.max_successive_merges: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.ttl: 2592000 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T01:07:37.006 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.948+0000 7f7829250700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.949+0000 7f7829250700 4 rocksdb: DB pointer 0x564a5e1eb800 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.949+0000 7f78120e9700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.949+0000 7f78120e9700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-16T01:07:37.007 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:37.008 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T01:07:37.009 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.010 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-16T01:07:37.011 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: 2023-11-16T01:07:37.011 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.950+0000 7f7829250700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-16T01:07:37.011 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.950+0000 7f7829250700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-16T01:07:37.011 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph-mon: debug 2023-11-16T01:07:36.950+0000 7f7829250700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi133 for mon.smithi133 2023-11-16T01:07:37.314 INFO:teuthology.orchestra.run.smithi133.stderr:create mon.smithi133 on 2023-11-16T01:07:37.314 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:37.318 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl disable ceph.target 2023-11-16T01:07:37.325 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T01:07:37.407 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl enable ceph.target 2023-11-16T01:07:37.413 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T01:07:37.495 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl start ceph.target 2023-11-16T01:07:37.502 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl enable ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target 2023-11-16T01:07:37.508 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target. 2023-11-16T01:07:37.508 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target. 2023-11-16T01:07:37.584 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target 2023-11-16T01:07:37.590 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl daemon-reload 2023-11-16T01:07:37.673 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:07:37.696 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl reset-failed ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:07:37.701 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Failed to reset failed state of unit ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service: Unit ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service not loaded. 2023-11-16T01:07:37.702 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl enable ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:07:37.708 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target.wants/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service → /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T01:07:37.793 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:07:38.170 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-16T01:07:38.178 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: disabled 2023-11-16T01:07:38.179 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active firewalld.service 2023-11-16T01:07:38.186 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:07:38.186 INFO:teuthology.orchestra.run.smithi133.stderr:firewalld.service is not enabled 2023-11-16T01:07:38.186 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to enable service . firewalld.service is not available 2023-11-16T01:07:38.187 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-16T01:07:38.187 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mon to start... 2023-11-16T01:07:38.187 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mon... 2023-11-16T01:07:38.187 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/mon.smithi133:/var/lib/ceph/mon/ceph-smithi133:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-16T01:07:38.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:38 smithi133 bash[111353]: 6ceea01f448be8176641ee9488a0c49505c0459fe760da2dfe9fc24f27657efd 2023-11-16T01:07:38.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:38 smithi133 systemd[1]: Started Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7. 2023-11-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.586+0000 7f710f384700 1 Processor -- start 2023-11-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.586+0000 7f710f384700 1 -- start start 2023-11-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.586+0000 7f710f384700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.586+0000 7f710f384700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71080fb480 con 0x7f71080faae0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f710d120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f710d120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50722/0 (socket says 172.21.15.133:50722) 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f710d120700 1 -- 172.21.15.133:0/1089456791 learned_addr learned my addr 172.21.15.133:0/1089456791 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f710d120700 1 -- 172.21.15.133:0/1089456791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71080fb5c0 con 0x7f71080faae0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f710d120700 1 --2- 172.21.15.133:0/1089456791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70f800b5f0 tx=0x7f70f8006ee0).ready entity=mon.0 client_cookie=362c966079d79443 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f70f7fff700 1 -- 172.21.15.133:0/1089456791 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f800d600 con 0x7f71080faae0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f70f7fff700 1 -- 172.21.15.133:0/1089456791 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f800d760 con 0x7f71080faae0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.587+0000 7f70f7fff700 1 -- 172.21.15.133:0/1089456791 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f800da30 con 0x7f71080faae0 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 -- 172.21.15.133:0/1089456791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 msgr2=0x7f71080faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:38.590 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 --2- 172.21.15.133:0/1089456791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70f800b5f0 tx=0x7f70f8006ee0).stop 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 -- 172.21.15.133:0/1089456791 shutdown_connections 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 --2- 172.21.15.133:0/1089456791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f71080faf00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 -- 172.21.15.133:0/1089456791 >> 172.21.15.133:0/1089456791 conn(0x7f71080f6280 msgr2=0x7f71080f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 -- 172.21.15.133:0/1089456791 shutdown_connections 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.588+0000 7f710f384700 1 -- 172.21.15.133:0/1089456791 wait complete. 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710f384700 1 Processor -- start 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710f384700 1 -- start start 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710f384700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:38.593 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710f384700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71080ff0d0 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710d120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710d120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50736/0 (socket says 172.21.15.133:50736) 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710d120700 1 -- 172.21.15.133:0/101984092 learned_addr learned my addr 172.21.15.133:0/101984092 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710d120700 1 -- 172.21.15.133:0/101984092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f800b200 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f710d120700 1 --2- 172.21.15.133:0/101984092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f70f8006c70 tx=0x7f70f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.589+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f8011b40 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f8011ca0 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70f801d430 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71080ff270 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71080ff680 con 0x7f71080faae0 2023-11-16T01:07:38.594 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f70f800f070 con 0x7f71080faae0 2023-11-16T01:07:38.595 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.590+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f70f8027460 con 0x7f71080faae0 2023-11-16T01:07:38.595 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.591+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7108045ad0 con 0x7f71080faae0 2023-11-16T01:07:38.595 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.593+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f70f8016070 con 0x7f71080faae0 2023-11-16T01:07:38.662 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.660+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f71080ffde0 con 0x7f71080faae0 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.660+0000 7f70f5ffb700 1 -- 172.21.15.133:0/101984092 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f70f8036070 con 0x7f71080faae0 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: cluster: 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: id: 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: services: 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi133 (age 0.504446s) 2023-11-16T01:07:38.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: data: 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: pgs: 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 msgr2=0x7f7108100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 --2- 172.21.15.133:0/101984092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f70f8006c70 tx=0x7f70f8016000).stop 2023-11-16T01:07:38.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 shutdown_connections 2023-11-16T01:07:38.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 --2- 172.21.15.133:0/101984092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71080faae0 0x7f7108100c50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:38.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 >> 172.21.15.133:0/101984092 conn(0x7f71080f6280 msgr2=0x7f71080f6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:38.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 shutdown_connections 2023-11-16T01:07:38.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:38.662+0000 7f710f384700 1 -- 172.21.15.133:0/101984092 wait complete. 2023-11-16T01:07:39.018 INFO:teuthology.orchestra.run.smithi133.stderr:mon is available 2023-11-16T01:07:39.019 INFO:teuthology.orchestra.run.smithi133.stderr:Assimilating anything we can from ceph.conf... 2023-11-16T01:07:39.019 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z -v /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/mon.smithi133:/var/lib/ceph/mon/ceph-smithi133:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi133/config 2023-11-16T01:07:39.236 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: cluster 2023-11-16T01:07:38.157706+0000 mon.smithi133 (mon.0) 2 : cluster [INF] mon.smithi133 is new leader, mons smithi133 in quorum (ranks 0) 2023-11-16T01:07:39.236 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: cluster 2023-11-16T01:07:38.157875+0000 mon.smithi133 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi133=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} 2023-11-16T01:07:39.236 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: cluster 2023-11-16T01:07:38.160527+0000 mon.smithi133 (mon.0) 4 : cluster [DBG] fsmap 2023-11-16T01:07:39.237 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: cluster 2023-11-16T01:07:38.163087+0000 mon.smithi133 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-16T01:07:39.237 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: cluster 2023-11-16T01:07:38.163556+0000 mon.smithi133 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-16T01:07:39.237 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: audit 2023-11-16T01:07:38.662116+0000 mon.smithi133 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.133:0/101984092' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T01:07:39.419 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2dfaa700 1 Processor -- start 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2dfaa700 1 -- start start 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2dfaa700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2dfaa700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280fc430 con 0x7f1e2806fbf0 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2cfa8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2cfa8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50742/0 (socket says 172.21.15.133:50742) 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2cfa8700 1 -- 172.21.15.133:0/3409829258 learned_addr learned my addr 172.21.15.133:0/3409829258 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.417+0000 7f1e2cfa8700 1 -- 172.21.15.133:0/3409829258 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28100f30 con 0x7f1e2806fbf0 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2cfa8700 1 --2- 172.21.15.133:0/3409829258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1e1800b5f0 tx=0x7f1e18006ee0).ready entity=mon.0 client_cookie=38dee70c80aaecbf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e277fe700 1 -- 172.21.15.133:0/3409829258 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1800d600 con 0x7f1e2806fbf0 2023-11-16T01:07:39.420 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e277fe700 1 -- 172.21.15.133:0/3409829258 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1800d760 con 0x7f1e2806fbf0 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e277fe700 1 -- 172.21.15.133:0/3409829258 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1800da30 con 0x7f1e2806fbf0 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3409829258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 msgr2=0x7f1e28100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 --2- 172.21.15.133:0/3409829258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1e1800b5f0 tx=0x7f1e18006ee0).stop 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3409829258 shutdown_connections 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 --2- 172.21.15.133:0/3409829258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e28100a40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3409829258 >> 172.21.15.133:0/3409829258 conn(0x7f1e280f60e0 msgr2=0x7f1e280f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3409829258 shutdown_connections 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.418+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3409829258 wait complete. 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2dfaa700 1 Processor -- start 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2dfaa700 1 -- start start 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2dfaa700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2dfaa700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2810c720 con 0x7f1e2806fbf0 2023-11-16T01:07:39.424 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2cfa8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2cfa8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50746/0 (socket says 172.21.15.133:50746) 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.419+0000 7f1e2cfa8700 1 -- 172.21.15.133:0/3885834164 learned_addr learned my addr 172.21.15.133:0/3885834164 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e2cfa8700 1 -- 172.21.15.133:0/3885834164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1800b200 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e2cfa8700 1 --2- 172.21.15.133:0/3885834164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1e18006c70 tx=0x7f1e18016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e18011c80 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18011de0 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1801d5d0 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2810c920 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2810cd30 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1e1800f070 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.420+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1e18027460 con 0x7f1e2806fbf0 2023-11-16T01:07:39.425 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.421+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28059100 con 0x7f1e2806fbf0 2023-11-16T01:07:39.426 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.423+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1e18016070 con 0x7f1e2806fbf0 2023-11-16T01:07:39.489 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.487+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1e28045ad0 con 0x7f1e2806fbf0 2023-11-16T01:07:39.494 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.491+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1e18027b60 con 0x7f1e2806fbf0 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.491+0000 7f1e257fa700 1 -- 172.21.15.133:0/3885834164 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f1e18031af0 con 0x7f1e2806fbf0 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: [global] 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: fsid = 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.133:3300,v1:172.21.15.133:6789] 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: [mgr] 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.495 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: [osd] 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.491+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 msgr2=0x7f1e2810c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.491+0000 7f1e2dfaa700 1 --2- 172.21.15.133:0/3885834164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1e18006c70 tx=0x7f1e18016000).stop 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.492+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 shutdown_connections 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.492+0000 7f1e2dfaa700 1 --2- 172.21.15.133:0/3885834164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2806fbf0 0x7f1e2810c230 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.492+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 >> 172.21.15.133:0/3885834164 conn(0x7f1e280f60e0 msgr2=0x7f1e280f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.492+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 shutdown_connections 2023-11-16T01:07:39.496 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:39.492+0000 7f1e2dfaa700 1 -- 172.21.15.133:0/3885834164 wait complete. 2023-11-16T01:07:39.842 INFO:teuthology.orchestra.run.smithi133.stderr:Generating new minimal ceph.conf... 2023-11-16T01:07:39.842 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z -v /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/mon.smithi133:/var/lib/ceph/mon/ceph-smithi133:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi133/config 2023-11-16T01:07:40.223 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.218+0000 7f336f7e2700 1 Processor -- start 2023-11-16T01:07:40.223 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336f7e2700 1 -- start start 2023-11-16T01:07:40.223 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336f7e2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336f7e2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33680fed20 con 0x7f336806fcd0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336d57e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336d57e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50760/0 (socket says 172.21.15.133:50760) 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336d57e700 1 -- 172.21.15.133:0/212243592 learned_addr learned my addr 172.21.15.133:0/212243592 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336d57e700 1 -- 172.21.15.133:0/212243592 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680fee60 con 0x7f336806fcd0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.219+0000 7f336d57e700 1 --2- 172.21.15.133:0/212243592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f335800b5f0 tx=0x7f3358006ee0).ready entity=mon.0 client_cookie=bff7551f8e10cd5a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f3357fff700 1 -- 172.21.15.133:0/212243592 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335800d600 con 0x7f336806fcd0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f3357fff700 1 -- 172.21.15.133:0/212243592 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f335800dbd0 con 0x7f336806fcd0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f3357fff700 1 -- 172.21.15.133:0/212243592 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3358011c00 con 0x7f336806fcd0 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 -- 172.21.15.133:0/212243592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 msgr2=0x7f33680fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 --2- 172.21.15.133:0/212243592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f335800b5f0 tx=0x7f3358006ee0).stop 2023-11-16T01:07:40.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 -- 172.21.15.133:0/212243592 shutdown_connections 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 --2- 172.21.15.133:0/212243592 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f33680fe7a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 -- 172.21.15.133:0/212243592 >> 172.21.15.133:0/212243592 conn(0x7f33680f6280 msgr2=0x7f33680f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 -- 172.21.15.133:0/212243592 shutdown_connections 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.220+0000 7f336f7e2700 1 -- 172.21.15.133:0/212243592 wait complete. 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336f7e2700 1 Processor -- start 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336f7e2700 1 -- start start 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336f7e2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336f7e2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336810ae90 con 0x7f336806fcd0 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336d57e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336d57e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50776/0 (socket says 172.21.15.133:50776) 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336d57e700 1 -- 172.21.15.133:0/2016411061 learned_addr learned my addr 172.21.15.133:0/2016411061 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336d57e700 1 -- 172.21.15.133:0/2016411061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335800b200 con 0x7f336806fcd0 2023-11-16T01:07:40.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.221+0000 7f336d57e700 1 --2- 172.21.15.133:0/2016411061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f335800e040 tx=0x7f3358006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:40.226 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335801b410 con 0x7f336806fcd0 2023-11-16T01:07:40.226 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f335801b9e0 con 0x7f336806fcd0 2023-11-16T01:07:40.226 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336810b090 con 0x7f336806fcd0 2023-11-16T01:07:40.226 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335801a770 con 0x7f336806fcd0 2023-11-16T01:07:40.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336810b4a0 con 0x7f336806fcd0 2023-11-16T01:07:40.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f335800f070 con 0x7f336806fcd0 2023-11-16T01:07:40.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.222+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3358024a80 con 0x7f336806fcd0 2023-11-16T01:07:40.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.223+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3368045ad0 con 0x7f336806fcd0 2023-11-16T01:07:40.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.225+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3358016070 con 0x7f336806fcd0 2023-11-16T01:07:40.290 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.289+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3368104820 con 0x7f336806fcd0 2023-11-16T01:07:40.291 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.289+0000 7f3355ffb700 1 -- 172.21.15.133:0/2016411061 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f3358006a70 con 0x7f336806fcd0 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 msgr2=0x7f336810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 --2- 172.21.15.133:0/2016411061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f335800e040 tx=0x7f3358006ee0).stop 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 shutdown_connections 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 --2- 172.21.15.133:0/2016411061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f336806fcd0 0x7f336810a9a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 >> 172.21.15.133:0/2016411061 conn(0x7f33680f6280 msgr2=0x7f33680f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 shutdown_connections 2023-11-16T01:07:40.292 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:40.290+0000 7f336f7e2700 1 -- 172.21.15.133:0/2016411061 wait complete. 2023-11-16T01:07:40.466 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: audit 2023-11-16T01:07:39.489079+0000 mon.smithi133 (mon.0) 8 : audit [INF] from='client.? 172.21.15.133:0/3885834164' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T01:07:40.466 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: audit 2023-11-16T01:07:39.492421+0000 mon.smithi133 (mon.0) 9 : audit [INF] from='client.? 172.21.15.133:0/3885834164' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T01:07:40.650 INFO:teuthology.orchestra.run.smithi133.stderr:Restarting the monitor... 2023-11-16T01:07:40.651 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl restart ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:07:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:40 smithi133 systemd[1]: Stopping Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7... 2023-11-16T01:07:41.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: debug 2023-11-16T01:07:40.740+0000 7f3f7b14d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi133 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-16T01:07:41.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111367]: debug 2023-11-16T01:07:40.740+0000 7f3f7b14d700 -1 mon.smithi133@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T01:07:41.523 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 podman[111690]: ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133 2023-11-16T01:07:41.525 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 systemd[1]: ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service: Succeeded. 2023-11-16T01:07:41.525 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 systemd[1]: Stopped Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7. 2023-11-16T01:07:41.525 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 systemd[1]: Starting Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7... 2023-11-16T01:07:41.601 INFO:teuthology.orchestra.run.smithi133.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-16T01:07:41.601 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.985+0000 7f5839f8f700 1 Processor -- start 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5839f8f700 1 -- start start 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5839f8f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5839f8f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340feb80 con 0x7f583406fbf0 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5838f8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5838f8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50786/0 (socket says 172.21.15.133:50786) 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.986+0000 7f5838f8d700 1 -- 172.21.15.133:0/3418203638 learned_addr learned my addr 172.21.15.133:0/3418203638 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:41.990 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5838f8d700 1 -- 172.21.15.133:0/3418203638 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340fecc0 con 0x7f583406fbf0 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5838f8d700 1 --2- 172.21.15.133:0/3418203638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f582800b5f0 tx=0x7f5828006ee0).ready entity=mon.0 client_cookie=fe205a7c03ab72a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f58337fe700 1 -- 172.21.15.133:0/3418203638 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582800d600 con 0x7f583406fbf0 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f58337fe700 1 -- 172.21.15.133:0/3418203638 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f582800dbd0 con 0x7f583406fbf0 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f58337fe700 1 -- 172.21.15.133:0/3418203638 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5828011c30 con 0x7f583406fbf0 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5839f8f700 1 -- 172.21.15.133:0/3418203638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 msgr2=0x7f58340fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5839f8f700 1 --2- 172.21.15.133:0/3418203638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f582800b5f0 tx=0x7f5828006ee0).stop 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5839f8f700 1 -- 172.21.15.133:0/3418203638 shutdown_connections 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5839f8f700 1 --2- 172.21.15.133:0/3418203638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f58340fe690 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.987+0000 7f5839f8f700 1 -- 172.21.15.133:0/3418203638 >> 172.21.15.133:0/3418203638 conn(0x7f58340f6130 msgr2=0x7f58340f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.988+0000 7f5839f8f700 1 -- 172.21.15.133:0/3418203638 shutdown_connections 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.988+0000 7f5839f8f700 1 -- 172.21.15.133:0/3418203638 wait complete. 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.988+0000 7f5839f8f700 1 Processor -- start 2023-11-16T01:07:41.991 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.988+0000 7f5839f8f700 1 -- start start 2023-11-16T01:07:41.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5839f8f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:41.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5839f8f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410ace0 con 0x7f583406fbf0 2023-11-16T01:07:41.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5838f8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:41.993 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5838f8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50788/0 (socket says 172.21.15.133:50788) 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5838f8d700 1 -- 172.21.15.133:0/4096467190 learned_addr learned my addr 172.21.15.133:0/4096467190 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5838f8d700 1 -- 172.21.15.133:0/4096467190 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582800b200 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5838f8d700 1 --2- 172.21.15.133:0/4096467190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5828011550 tx=0x7f5828006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582800d9f0 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.989+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583410aee0 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f582801a9f0 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582801ba30 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583410b2f0 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f582800f070 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5828025cd0 con 0x7f583406fbf0 2023-11-16T01:07:41.994 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.990+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834045ad0 con 0x7f583406fbf0 2023-11-16T01:07:41.995 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:41.993+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5828016070 con 0x7f583406fbf0 2023-11-16T01:07:42.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.591+0000 7f4d73ad3700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-16T01:07:42.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.591+0000 7f4d73ad3700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-16T01:07:42.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.591+0000 7f4d73ad3700 0 pidfile_write: ignore empty --pid-file 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 0 load: jerasure load: lrc load: isa 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: RocksDB version: 6.8.1 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Compile date May 13 2021 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: DB SUMMARY 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: CURRENT file: CURRENT 2023-11-16T01:07:42.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi133/store.db dir, Total Num: 1, files: 000004.sst 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi133/store.db: 000006.log size: 92472 ; 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.error_if_exists: 0 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 bash[111762]: 58c0e7d081c600bf6af04b875c12cc1e86f50730a05dfc576a9ba9fb64885116 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.create_if_missing: 0 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.paranoid_checks: 1 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.env: 0x5617258f21c0 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.fs: Posix File System 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.info_log: 0x56172672f960 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.statistics: (nil) 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.use_fsync: 0 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.595+0000 7f4d73ad3700 4 rocksdb: Options.max_log_file_size: 0 2023-11-16T01:07:42.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.allow_fallocate: 1 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.use_direct_reads: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T01:07:42.036 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.db_log_dir: 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi133/store.db 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_subcompactions: 1 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_background_flushes: -1 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T01:07:42.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.write_buffer_manager: 0x561726735fb0 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-16T01:07:42.038 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T01:07:42.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-16T01:07:42.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-16T01:07:42.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-16T01:07:42.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.unordered_write: 0 2023-11-16T01:07:42.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.row_cache: None 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.wal_filter: None 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T01:07:42.040 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.preserve_deletes: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.two_write_queues: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.atomic_flush: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.log_readahead_size: 0 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-16T01:07:42.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_background_jobs: 2 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_background_compactions: -1 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T01:07:42.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_open_files: -1 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Compression algorithms supported: 2023-11-16T01:07:42.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kZSTD supported: 0 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kXpressCompression supported: 0 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kLZ4Compression supported: 1 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kBZip2Compression supported: 0 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kZlibCompression supported: 1 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: kSnappyCompression supported: 1 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000005 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.merge_operator: 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_filter: None 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561726646128) 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cache_index_and_filter_blocks: 1 2023-11-16T01:07:42.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: pin_top_level_index_and_filter: 1 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: index_type: 0 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: data_block_index_type: 0 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: index_shortening: 1 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: data_block_hash_table_util_ratio: 0.750000 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: hash_index_allow_collision: 1 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: checksum: 1 2023-11-16T01:07:42.046 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: no_block_cache: 0 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_cache: 0x56172667cf10 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_cache_name: BinnedLRUCache 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_cache_options: 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: capacity : 536870912 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: num_shard_bits : 4 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: strict_capacity_limit : 0 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: high_pri_pool_ratio: 0.000 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_cache_compressed: (nil) 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: persistent_cache: (nil) 2023-11-16T01:07:42.047 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_size: 4096 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_size_deviation: 10 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_restart_interval: 16 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: index_block_restart_interval: 1 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: metadata_block_size: 4096 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: partition_filters: 0 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: use_delta_encoding: 1 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: whole_key_filtering: 1 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: verify_compression: 0 2023-11-16T01:07:42.048 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: read_amp_bytes_per_bit: 0 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: format_version: 2 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: enable_index_compression: 1 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: block_align: 0 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression: NoCompression 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-16T01:07:42.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T01:07:42.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.num_levels: 7 2023-11-16T01:07:42.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T01:07:42.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T01:07:42.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T01:07:42.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T01:07:42.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T01:07:42.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T01:07:42.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T01:07:42.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.table_properties_collectors: 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.inplace_update_support: 0 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T01:07:42.055 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.bloom_locality: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.max_successive_merges: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.ttl: 2592000 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T01:07:42.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700096861598225, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.596+0000 7f4d73ad3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.597+0000 7f4d73ad3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.597+0000 7f4d73ad3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700096861599093, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88272, "table_properties": {"data_size": 86399, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79613, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700096861, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.597+0000 7f4d73ad3700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.057 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 systemd[1]: Started Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7. 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.599+0000 7f4d73ad3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700096861600618, "job": 1, "event": "recovery_finished"} 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.599+0000 7f4d73ad3700 4 rocksdb: DB pointer 0x561726741800 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.600+0000 7f4d5a8eb700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.600+0000 7f4d5a8eb700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** DB Stats ** 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T01:07:42.058 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** Compaction Stats [default] ** 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: L0 2/0 87.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Sum 2/0 87.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** Compaction Stats [default] ** 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T01:07:42.060 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(Keys): cumulative 0, interval 0 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Cumulative compaction: 0.00 GB write, 21.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Interval compaction: 0.00 GB write, 21.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** File Read Latency Histogram By Level [default] ** 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** Compaction Stats [default] ** 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:42.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: L0 2/0 87.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Sum 2/0 87.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** Compaction Stats [default] ** 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:42.062 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: AddFile(Keys): cumulative 0, interval 0 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Cumulative compaction: 0.00 GB write, 21.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ** File Read Latency Histogram By Level [default] ** 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.600+0000 7f4d73ad3700 0 starting mon.smithi133 rank 0 at public addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] at bind addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi133 fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:42.063 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 1 mon.smithi133@-1(???) e1 preinit fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).mds e1 new map 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).mds e1 print_map 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: e1 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: legacy client fscid: -1 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: No filesystems configured 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T01:07:42.064 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T01:07:42.065 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T01:07:42.065 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 0 mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:39.489079+0000 mon.smithi133 (mon.0) 8 : audit [INF] from='client.? 172.21.15.133:0/3885834164' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:39.492421+0000 mon.smithi133 (mon.0) 9 : audit [INF] from='client.? 172.21.15.133:0/3885834164' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.602+0000 7f4d73ad3700 1 mon.smithi133@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expand map: {default=false} 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta from 'false' to 'false' 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expanded map: {default=false} 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expand map: {default=info} 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta from 'info' to 'info' 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expanded map: {default=info} 2023-11-16T01:07:42.066 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expand map: {default=daemon} 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expand map: {default=debug} 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:07:41.603+0000 7f4d73ad3700 20 expand_channel_meta expanded map: {default=debug} 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:41.619358+0000 mon.smithi133 (mon.0) 1 : cluster [INF] mon.smithi133 is new leader, mons smithi133 in quorum (ranks 0) 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:41.619447+0000 mon.smithi133 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi133=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:41.619499+0000 mon.smithi133 (mon.0) 3 : cluster [DBG] fsmap 2023-11-16T01:07:42.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:41.619540+0000 mon.smithi133 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-16T01:07:42.068 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:41.620074+0000 mon.smithi133 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.056+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f58341046a0 con 0x7f583406fbf0 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.061+0000 7f58317fa700 1 -- 172.21.15.133:0/4096467190 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5828025460 con 0x7f583406fbf0 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 msgr2=0x7f583410a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 --2- 172.21.15.133:0/4096467190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5828011550 tx=0x7f5828006ee0).stop 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 shutdown_connections 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 --2- 172.21.15.133:0/4096467190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f583406fbf0 0x7f583410a7f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 >> 172.21.15.133:0/4096467190 conn(0x7f58340f6130 msgr2=0x7f58340f7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:42.069 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 shutdown_connections 2023-11-16T01:07:42.070 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:42.062+0000 7f5839f8f700 1 -- 172.21.15.133:0/4096467190 wait complete. 2023-11-16T01:07:42.427 INFO:teuthology.orchestra.run.smithi133.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-16T01:07:42.428 INFO:teuthology.orchestra.run.smithi133.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T01:07:42.428 INFO:teuthology.orchestra.run.smithi133.stderr:Creating mgr... 2023-11-16T01:07:42.428 INFO:teuthology.orchestra.run.smithi133.stderr:Verifying port 9283 ... 2023-11-16T01:07:42.429 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl daemon-reload 2023-11-16T01:07:42.525 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl 2023-11-16T01:07:42.552 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl reset-failed ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl 2023-11-16T01:07:42.557 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Failed to reset failed state of unit ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl.service: Unit ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl.service not loaded. 2023-11-16T01:07:42.558 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl enable ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl 2023-11-16T01:07:42.563 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7.target.wants/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl.service → /etc/systemd/system/ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T01:07:42.647 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mgr.smithi133.xdyttl 2023-11-16T01:07:42.993 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-16T01:07:43.001 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: disabled 2023-11-16T01:07:43.001 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active firewalld.service 2023-11-16T01:07:43.008 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:07:43.009 INFO:teuthology.orchestra.run.smithi133.stderr:firewalld.service is not enabled 2023-11-16T01:07:43.009 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to enable service . firewalld.service is not available 2023-11-16T01:07:43.009 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-16T01:07:43.009 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-16T01:07:43.017 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: disabled 2023-11-16T01:07:43.017 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active firewalld.service 2023-11-16T01:07:43.024 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:07:43.026 INFO:teuthology.orchestra.run.smithi133.stderr:firewalld.service is not enabled 2023-11-16T01:07:43.026 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-16T01:07:43.027 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mgr to start... 2023-11-16T01:07:43.027 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mgr... 2023-11-16T01:07:43.027 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-16T01:07:43.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:42.063050+0000 mon.smithi133 (mon.0) 6 : audit [INF] from='client.? 172.21.15.133:0/4096467190' entity='client.admin' 2023-11-16T01:07:43.451 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.449+0000 7fb7f9a70700 1 Processor -- start 2023-11-16T01:07:43.453 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.449+0000 7fb7f9a70700 1 -- start start 2023-11-16T01:07:43.453 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.449+0000 7fb7f9a70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:43.453 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.449+0000 7fb7f9a70700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40fa870 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.453 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:43.453 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54076/0 (socket says 172.21.15.133:54076) 2023-11-16T01:07:43.456 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f37fe700 1 -- 172.21.15.133:0/922609563 learned_addr learned my addr 172.21.15.133:0/922609563 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:43.456 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f37fe700 1 -- 172.21.15.133:0/922609563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f40fa9b0 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.456 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f37fe700 1 --2- 172.21.15.133:0/922609563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb7e400b5f0 tx=0x7fb7e4006ee0).ready entity=mon.0 client_cookie=e086f1adf4b15d18 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:43.456 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f27fc700 1 -- 172.21.15.133:0/922609563 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7e400d600 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f27fc700 1 -- 172.21.15.133:0/922609563 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb7e400dbd0 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.450+0000 7fb7f27fc700 1 -- 172.21.15.133:0/922609563 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7e4011c00 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 -- 172.21.15.133:0/922609563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 msgr2=0x7fb7f40fa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 --2- 172.21.15.133:0/922609563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb7e400b5f0 tx=0x7fb7e4006ee0).stop 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 -- 172.21.15.133:0/922609563 shutdown_connections 2023-11-16T01:07:43.457 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 --2- 172.21.15.133:0/922609563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f40fa2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 -- 172.21.15.133:0/922609563 >> 172.21.15.133:0/922609563 conn(0x7fb7f40f59e0 msgr2=0x7fb7f40f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 -- 172.21.15.133:0/922609563 shutdown_connections 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.451+0000 7fb7f9a70700 1 -- 172.21.15.133:0/922609563 wait complete. 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f9a70700 1 Processor -- start 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f9a70700 1 -- start start 2023-11-16T01:07:43.458 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f9a70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f9a70700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f410af40 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54078/0 (socket says 172.21.15.133:54078) 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f37fe700 1 -- 172.21.15.133:0/2910331018 learned_addr learned my addr 172.21.15.133:0/2910331018 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.452+0000 7fb7f37fe700 1 -- 172.21.15.133:0/2910331018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e400b200 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7f37fe700 1 --2- 172.21.15.133:0/2910331018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb7e4002f40 tx=0x7fb7e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:43.459 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7e4011690 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f410b140 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb7e401b880 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7e401a6b0 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f410b550 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb7e400f070 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.460 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.453+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb7e4025a00 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.461 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.454+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f4045ad0 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.461 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.458+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb7e4016070 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.530 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.526+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb7f41048b0 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.530 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.527+0000 7fb7dbfff700 1 -- 172.21.15.133:0/2910331018 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fb7e401ad20 con 0x7fb7f40f9ed0 2023-11-16T01:07:43.530 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.530 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsid": "7905ec40-841c-11ee-8dbe-212e2dc638e7", 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "health": { 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 0 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T01:07:43.531 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "smithi133" 2023-11-16T01:07:43.532 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:43.532 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-16T01:07:43.532 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T01:07:43.532 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:43.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T01:07:43.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T01:07:43.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T01:07:43.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:43.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "available": false, 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "iostat", 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "restful" 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T01:07:43.542 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modified": "2023-11-16T01:07:38.159164+0000", 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 msgr2=0x7fb7f410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 --2- 172.21.15.133:0/2910331018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb7e4002f40 tx=0x7fb7e4006ee0).stop 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 shutdown_connections 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 --2- 172.21.15.133:0/2910331018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7f40f9ed0 0x7fb7f410aa50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 >> 172.21.15.133:0/2910331018 conn(0x7fb7f40f59e0 msgr2=0x7fb7f40f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 shutdown_connections 2023-11-16T01:07:43.543 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:43.528+0000 7fb7f9a70700 1 -- 172.21.15.133:0/2910331018 wait complete. 2023-11-16T01:07:43.908 INFO:teuthology.orchestra.run.smithi133.stderr:mgr not available, waiting (1/15)... 2023-11-16T01:07:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:44 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:43.528376+0000 mon.smithi133 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.133:0/2910331018' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T01:07:45.911 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.321+0000 7f20735c6700 1 Processor -- start 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.321+0000 7f20735c6700 1 -- start start 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f20735c6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f20735c6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0fc6a0 con 0x7f206c06fc50 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f2071362700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:46.324 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f2071362700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54080/0 (socket says 172.21.15.133:54080) 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f2071362700 1 -- 172.21.15.133:0/612593911 learned_addr learned my addr 172.21.15.133:0/612593911 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f2071362700 1 -- 172.21.15.133:0/612593911 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c101110 con 0x7f206c06fc50 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f2071362700 1 --2- 172.21.15.133:0/612593911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f206800b5f0 tx=0x7f2068006ee0).ready entity=mon.0 client_cookie=9a4851a41ab130da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f205bfff700 1 -- 172.21.15.133:0/612593911 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f206800d600 con 0x7f206c06fc50 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f205bfff700 1 -- 172.21.15.133:0/612593911 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f206800dbd0 con 0x7f206c06fc50 2023-11-16T01:07:46.325 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.322+0000 7f205bfff700 1 -- 172.21.15.133:0/612593911 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068011c00 con 0x7f206c06fc50 2023-11-16T01:07:46.328 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 -- 172.21.15.133:0/612593911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 msgr2=0x7f206c100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 --2- 172.21.15.133:0/612593911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f206800b5f0 tx=0x7f2068006ee0).stop 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 -- 172.21.15.133:0/612593911 shutdown_connections 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 --2- 172.21.15.133:0/612593911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c100c20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 -- 172.21.15.133:0/612593911 >> 172.21.15.133:0/612593911 conn(0x7f206c0f6280 msgr2=0x7f206c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 -- 172.21.15.133:0/612593911 shutdown_connections 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.323+0000 7f20735c6700 1 -- 172.21.15.133:0/612593911 wait complete. 2023-11-16T01:07:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f20735c6700 1 Processor -- start 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f20735c6700 1 -- start start 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f20735c6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f20735c6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c10d0c0 con 0x7f206c06fc50 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f2071362700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f2071362700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54086/0 (socket says 172.21.15.133:54086) 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f2071362700 1 -- 172.21.15.133:0/1541558043 learned_addr learned my addr 172.21.15.133:0/1541558043 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f2071362700 1 -- 172.21.15.133:0/1541558043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206800b200 con 0x7f206c06fc50 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.324+0000 7f2071362700 1 --2- 172.21.15.133:0/1541558043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f206800b1d0 tx=0x7f2068006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068011690 con 0x7f206c06fc50 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c10d2c0 con 0x7f206c06fc50 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f206801b880 con 0x7f206c06fc50 2023-11-16T01:07:46.330 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f206801a650 con 0x7f206c06fc50 2023-11-16T01:07:46.331 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c10d6d0 con 0x7f206c06fc50 2023-11-16T01:07:46.331 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f206800f070 con 0x7f206c06fc50 2023-11-16T01:07:46.331 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.325+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2068025b10 con 0x7f206c06fc50 2023-11-16T01:07:46.331 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.326+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c045ad0 con 0x7f206c06fc50 2023-11-16T01:07:46.331 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.328+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2068016070 con 0x7f206c06fc50 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.396+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f206c106a50 con 0x7f206c06fc50 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.397+0000 7f2059ffb700 1 -- 172.21.15.133:0/1541558043 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f206801bd40 con 0x7f206c06fc50 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsid": "7905ec40-841c-11ee-8dbe-212e2dc638e7", 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "health": { 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.404 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 0 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "smithi133" 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T01:07:46.405 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T01:07:46.406 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T01:07:46.407 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "available": false, 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "iostat", 2023-11-16T01:07:46.408 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "restful" 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modified": "2023-11-16T01:07:38.159164+0000", 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 msgr2=0x7f206c10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 --2- 172.21.15.133:0/1541558043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f206800b1d0 tx=0x7f2068006ee0).stop 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 shutdown_connections 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 --2- 172.21.15.133:0/1541558043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f206c06fc50 0x7f206c10cbd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:46.409 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 >> 172.21.15.133:0/1541558043 conn(0x7f206c0f6280 msgr2=0x7f206c0f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:46.410 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 shutdown_connections 2023-11-16T01:07:46.410 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:46.398+0000 7f20735c6700 1 -- 172.21.15.133:0/1541558043 wait complete. 2023-11-16T01:07:46.701 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:46 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:46.398262+0000 mon.smithi133 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.133:0/1541558043' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T01:07:46.766 INFO:teuthology.orchestra.run.smithi133.stderr:mgr not available, waiting (2/15)... 2023-11-16T01:07:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:47.948509+0000 mon.smithi133 (mon.0) 9 : cluster [INF] Activating manager daemon smithi133.xdyttl 2023-11-16T01:07:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:47.951140+0000 mon.smithi133 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi133.xdyttl(active, starting, since 0.00274028s) 2023-11-16T01:07:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.952254+0000 mon.smithi133 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.952415+0000 mon.smithi133 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.952532+0000 mon.smithi133 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.952672+0000 mon.smithi133 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.952827+0000 mon.smithi133 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mgr metadata", "who": "smithi133.xdyttl", "id": "smithi133.xdyttl"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:47.966847+0000 mon.smithi133 (mon.0) 16 : cluster [INF] Manager daemon smithi133.xdyttl is now available 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.974383+0000 mon.smithi133 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.978150+0000 mon.smithi133 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.979270+0000 mon.smithi133 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.983957+0000 mon.smithi133 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:47.985813+0000 mon.smithi133 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.133:0/2650325562' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:48.769 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-16T01:07:49.162 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec2f53700 1 Processor -- start 2023-11-16T01:07:49.162 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec2f53700 1 -- start start 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec2f53700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec2f53700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0fc660 con 0x7f2ebc06fc90 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec0cef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec0cef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54158/0 (socket says 172.21.15.133:54158) 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec0cef700 1 -- 172.21.15.133:0/897221218 learned_addr learned my addr 172.21.15.133:0/897221218 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.159+0000 7f2ec0cef700 1 -- 172.21.15.133:0/897221218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc1010d0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec0cef700 1 --2- 172.21.15.133:0/897221218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2eb000b5f0 tx=0x7f2eb0006ee0).ready entity=mon.0 client_cookie=eecea08d43e656d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ebb7fe700 1 -- 172.21.15.133:0/897221218 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb000d600 con 0x7f2ebc06fc90 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ebb7fe700 1 -- 172.21.15.133:0/897221218 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2eb000dbd0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ebb7fe700 1 -- 172.21.15.133:0/897221218 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb0011c00 con 0x7f2ebc06fc90 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 -- 172.21.15.133:0/897221218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 msgr2=0x7f2ebc100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/897221218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2eb000b5f0 tx=0x7f2eb0006ee0).stop 2023-11-16T01:07:49.163 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 -- 172.21.15.133:0/897221218 shutdown_connections 2023-11-16T01:07:49.164 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/897221218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc100be0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:49.164 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 -- 172.21.15.133:0/897221218 >> 172.21.15.133:0/897221218 conn(0x7f2ebc0f6280 msgr2=0x7f2ebc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:49.164 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 -- 172.21.15.133:0/897221218 shutdown_connections 2023-11-16T01:07:49.164 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.160+0000 7f2ec2f53700 1 -- 172.21.15.133:0/897221218 wait complete. 2023-11-16T01:07:49.166 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.161+0000 7f2ec2f53700 1 Processor -- start 2023-11-16T01:07:49.166 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.161+0000 7f2ec2f53700 1 -- start start 2023-11-16T01:07:49.166 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec2f53700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:49.166 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec2f53700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc10d0b0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.166 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec0cef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec0cef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54172/0 (socket says 172.21.15.133:54172) 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec0cef700 1 -- 172.21.15.133:0/2042379201 learned_addr learned my addr 172.21.15.133:0/2042379201 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec0cef700 1 -- 172.21.15.133:0/2042379201 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb000b200 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec0cef700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2eb000e040 tx=0x7f2eb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb001b410 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2eb001b9e0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb001a770 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc10d2b0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.162+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc10d6c0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.163+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f2eb000f070 con 0x7f2ebc06fc90 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.163+0000 7f2eb97fa700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 0x7f2ea0032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:49.167 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.163+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2eb00417c0 con 0x7f2ebc06fc90 2023-11-16T01:07:49.168 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.163+0000 7f2ebbfff700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 0x7f2ea0032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:49.168 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.163+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc059100 con 0x7f2ebc06fc90 2023-11-16T01:07:49.168 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.164+0000 7f2ebbfff700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 0x7f2ea0032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2ea80096a0 tx=0x7f2ea8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:49.169 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.168+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2eb0016030 con 0x7f2ebc06fc90 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.369+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2ebc101e00 con 0x7f2ebc06fc90 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.369+0000 7f2eb97fa700 1 -- 172.21.15.133:0/2042379201 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f2eb0024460 con 0x7f2ebc06fc90 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsid": "7905ec40-841c-11ee-8dbe-212e2dc638e7", 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "health": { 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "checks": {}, 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mutes": [] 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-16T01:07:49.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum": [ 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 0 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "smithi133" 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "monmap": { 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osdmap": { 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgmap": { 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-16T01:07:49.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "fsmap": { 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "available": true, 2023-11-16T01:07:49.380 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modules": [ 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "iostat", 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "restful" 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ], 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "servicemap": { 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "modified": "2023-11-16T01:07:38.159164+0000", 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "services": {} 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: }, 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-16T01:07:49.381 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 msgr2=0x7f2ea0032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 0x7f2ea0032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2ea80096a0 tx=0x7f2ea8006b40).stop 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 msgr2=0x7f2ebc10cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2eb000e040 tx=0x7f2eb0006ee0).stop 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 shutdown_connections 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f2ea002fca0 0x7f2ea0032160 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 --2- 172.21.15.133:0/2042379201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc06fc90 0x7f2ebc10cbc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 >> 172.21.15.133:0/2042379201 conn(0x7f2ebc0f6280 msgr2=0x7f2ebc0f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 shutdown_connections 2023-11-16T01:07:49.382 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:49.371+0000 7f2ec2f53700 1 -- 172.21.15.133:0/2042379201 wait complete. 2023-11-16T01:07:49.708 INFO:teuthology.orchestra.run.smithi133.stderr:mgr is available 2023-11-16T01:07:49.708 INFO:teuthology.orchestra.run.smithi133.stderr:Enabling cephadm module... 2023-11-16T01:07:49.708 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.078+0000 7f343d8c9700 1 Processor -- start 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.078+0000 7f343d8c9700 1 -- start start 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f343d8c9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f343d8c9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380fedd0 con 0x7f343806fc40 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3436ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3436ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54178/0 (socket says 172.21.15.133:54178) 2023-11-16T01:07:50.083 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3436ffd700 1 -- 172.21.15.133:0/3142157363 learned_addr learned my addr 172.21.15.133:0/3142157363 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3436ffd700 1 -- 172.21.15.133:0/3142157363 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380fef10 con 0x7f343806fc40 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3436ffd700 1 --2- 172.21.15.133:0/3142157363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f342000b5f0 tx=0x7f3420006ee0).ready entity=mon.0 client_cookie=b35b433f8bcdb405 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3435ffb700 1 -- 172.21.15.133:0/3142157363 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f342000d600 con 0x7f343806fc40 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3435ffb700 1 -- 172.21.15.133:0/3142157363 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f342000dbd0 con 0x7f343806fc40 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.079+0000 7f3435ffb700 1 -- 172.21.15.133:0/3142157363 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3420011c00 con 0x7f343806fc40 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 -- 172.21.15.133:0/3142157363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 msgr2=0x7f34380fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 --2- 172.21.15.133:0/3142157363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f342000b5f0 tx=0x7f3420006ee0).stop 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 -- 172.21.15.133:0/3142157363 shutdown_connections 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 --2- 172.21.15.133:0/3142157363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f34380fe8e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 -- 172.21.15.133:0/3142157363 >> 172.21.15.133:0/3142157363 conn(0x7f34380f62e0 msgr2=0x7f34380f8700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 -- 172.21.15.133:0/3142157363 shutdown_connections 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.080+0000 7f343d8c9700 1 -- 172.21.15.133:0/3142157363 wait complete. 2023-11-16T01:07:50.084 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f343d8c9700 1 Processor -- start 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f343d8c9700 1 -- start start 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f343d8c9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f343d8c9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343810af30 con 0x7f343806fc40 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f3436ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f3436ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54194/0 (socket says 172.21.15.133:54194) 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f3436ffd700 1 -- 172.21.15.133:0/2333719709 learned_addr learned my addr 172.21.15.133:0/2333719709 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:50.085 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f3436ffd700 1 -- 172.21.15.133:0/2333719709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342000b200 con 0x7f343806fc40 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.081+0000 7f3436ffd700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f342000b190 tx=0x7f3420006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f342001b410 con 0x7f343806fc40 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f342001b9e0 con 0x7f343806fc40 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f342001a880 con 0x7f343806fc40 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343810b130 con 0x7f343806fc40 2023-11-16T01:07:50.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343810b540 con 0x7f343806fc40 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.082+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f342000f070 con 0x7f343806fc40 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.083+0000 7f342f7fe700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 0x7f3418032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.083+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3420040ff0 con 0x7f343806fc40 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.083+0000 7f34367fc700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 0x7f3418032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.083+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3438059100 con 0x7f343806fc40 2023-11-16T01:07:50.087 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.083+0000 7f34367fc700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 0x7f3418032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f34280096a0 tx=0x7f3428006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:50.088 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.087+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3420025460 con 0x7f343806fc40 2023-11-16T01:07:50.272 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.271+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f3438104820 con 0x7f343806fc40 2023-11-16T01:07:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:48.953822+0000 mon.smithi133 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi133.xdyttl(active, since 1.00542s) 2023-11-16T01:07:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:49.370904+0000 mon.smithi133 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.133:0/2042379201' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T01:07:50.953 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.951+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f34200257e0 con 0x7f343806fc40 2023-11-16T01:07:50.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.953+0000 7f342f7fe700 1 -- 172.21.15.133:0/2333719709 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f3420022310 con 0x7f343806fc40 2023-11-16T01:07:50.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 msgr2=0x7f3418032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:50.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 0x7f3418032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f34280096a0 tx=0x7f3428006b40).stop 2023-11-16T01:07:50.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 msgr2=0x7f343810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f342000b190 tx=0x7f3420006ee0).stop 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 shutdown_connections 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f341802fca0 0x7f3418032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 --2- 172.21.15.133:0/2333719709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f343806fc40 0x7f343810aa40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 >> 172.21.15.133:0/2333719709 conn(0x7f34380f62e0 msgr2=0x7f34380f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 shutdown_connections 2023-11-16T01:07:50.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:50.955+0000 7f343d8c9700 1 -- 172.21.15.133:0/2333719709 wait complete. 2023-11-16T01:07:51.259 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:50.272779+0000 mon.smithi133 (mon.0) 24 : audit [INF] from='client.? 172.21.15.133:0/2333719709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T01:07:51.324 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04f094700 1 Processor -- start 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04f094700 1 -- start start 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04f094700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04f094700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480fc4e0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04ce30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04ce30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54230/0 (socket says 172.21.15.133:54230) 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.686+0000 7fc04ce30700 1 -- 172.21.15.133:0/1110817263 learned_addr learned my addr 172.21.15.133:0/1110817263 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.687+0000 7fc04ce30700 1 -- 172.21.15.133:0/1110817263 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480fc620 con 0x7fc0480fbbd0 2023-11-16T01:07:51.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.687+0000 7fc04ce30700 1 --2- 172.21.15.133:0/1110817263 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc03c00b5f0 tx=0x7fc03c006ee0).ready entity=mon.0 client_cookie=98514d33a20454d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:51.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.687+0000 7fc0477fe700 1 -- 172.21.15.133:0/1110817263 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c00d600 con 0x7fc0480fbbd0 2023-11-16T01:07:51.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.687+0000 7fc0477fe700 1 -- 172.21.15.133:0/1110817263 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc03c00dbd0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.687+0000 7fc0477fe700 1 -- 172.21.15.133:0/1110817263 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c011c00 con 0x7fc0480fbbd0 2023-11-16T01:07:51.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- 172.21.15.133:0/1110817263 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 msgr2=0x7fc0480fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:51.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 --2- 172.21.15.133:0/1110817263 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc03c00b5f0 tx=0x7fc03c006ee0).stop 2023-11-16T01:07:51.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- 172.21.15.133:0/1110817263 shutdown_connections 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 --2- 172.21.15.133:0/1110817263 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc0480fbff0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- 172.21.15.133:0/1110817263 >> 172.21.15.133:0/1110817263 conn(0x7fc0480f73f0 msgr2=0x7fc0480f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- 172.21.15.133:0/1110817263 shutdown_connections 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- 172.21.15.133:0/1110817263 wait complete. 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 Processor -- start 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.688+0000 7fc04f094700 1 -- start start 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04f094700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04f094700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480701f0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04ce30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04ce30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54242/0 (socket says 172.21.15.133:54242) 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04ce30700 1 -- 172.21.15.133:0/2953408563 learned_addr learned my addr 172.21.15.133:0/2953408563 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:51.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04ce30700 1 -- 172.21.15.133:0/2953408563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c00b200 con 0x7fc0480fbbd0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04ce30700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc03c00b1d0 tx=0x7fc03c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c01b410 con 0x7fc0480fbbd0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc03c01b9e0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.689+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc048070390 con 0x7fc0480fbbd0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.690+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c01a840 con 0x7fc0480fbbd0 2023-11-16T01:07:51.693 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.690+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0480707a0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.694 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.690+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc03c00f070 con 0x7fc0480fbbd0 2023-11-16T01:07:51.694 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.691+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc048045ad0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.694 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.691+0000 7fc0457fa700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 0x7fc02c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:51.694 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.691+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc03c041140 con 0x7fc0480fbbd0 2023-11-16T01:07:51.694 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.691+0000 7fc047fff700 1 -- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 msgr2=0x7fc02c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:51.695 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.691+0000 7fc047fff700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 0x7fc02c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T01:07:51.699 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.698+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc03c022d70 con 0x7fc0480fbbd0 2023-11-16T01:07:51.894 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.892+0000 7fc047fff700 1 -- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 msgr2=0x7fc02c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:51.894 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.892+0000 7fc047fff700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 0x7fc02c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T01:07:51.896 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.893+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc0480fd9e0 con 0x7fc0480fbbd0 2023-11-16T01:07:51.897 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.893+0000 7fc0457fa700 1 -- 172.21.15.133:0/2953408563 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc03c023e90 con 0x7fc0480fbbd0 2023-11-16T01:07:51.897 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:07:51.898 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 4, 2023-11-16T01:07:51.898 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "available": true, 2023-11-16T01:07:51.898 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "active_name": "smithi133.xdyttl", 2023-11-16T01:07:51.898 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-16T01:07:51.899 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:07:51.900 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 msgr2=0x7fc02c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 0x7fc02c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 msgr2=0x7fc048071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc03c00b1d0 tx=0x7fc03c006ee0).stop 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 shutdown_connections 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7fc02c02fdb0 0x7fc02c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:07:51.901 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 --2- 172.21.15.133:0/2953408563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0480fbbd0 0x7fc048071d70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:51.902 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 >> 172.21.15.133:0/2953408563 conn(0x7fc0480f73f0 msgr2=0x7fc0480f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:51.902 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 shutdown_connections 2023-11-16T01:07:51.902 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:51.896+0000 7fc04f094700 1 -- 172.21.15.133:0/2953408563 wait complete. 2023-11-16T01:07:52.248 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for the mgr to restart... 2023-11-16T01:07:52.248 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mgr epoch 4... 2023-11-16T01:07:52.248 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-16T01:07:52.253 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:50.954223+0000 mon.smithi133 (mon.0) 25 : audit [INF] from='client.? 172.21.15.133:0/2333719709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T01:07:52.253 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:50.954275+0000 mon.smithi133 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi133.xdyttl(active, since 3s) 2023-11-16T01:07:52.253 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:51.895076+0000 mon.smithi133 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.133:0/2953408563' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T01:07:52.625 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f4158f8a700 1 Processor -- start 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f4158f8a700 1 -- start start 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f4158f8a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f4158f8a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540fc6a0 con 0x7f415406fc50 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f415259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f415259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54256/0 (socket says 172.21.15.133:54256) 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.620+0000 7f415259c700 1 -- 172.21.15.133:0/3529884031 learned_addr learned my addr 172.21.15.133:0/3529884031 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.621+0000 7f415259c700 1 -- 172.21.15.133:0/3529884031 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154101110 con 0x7f415406fc50 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.621+0000 7f415259c700 1 --2- 172.21.15.133:0/3529884031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f414400b5f0 tx=0x7f4144006ee0).ready entity=mon.0 client_cookie=c3ebb1973e04f41a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:52.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.621+0000 7f415159a700 1 -- 172.21.15.133:0/3529884031 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414400d600 con 0x7f415406fc50 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.621+0000 7f415159a700 1 -- 172.21.15.133:0/3529884031 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f414400dbd0 con 0x7f415406fc50 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.621+0000 7f415159a700 1 -- 172.21.15.133:0/3529884031 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4144011c00 con 0x7f415406fc50 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- 172.21.15.133:0/3529884031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 msgr2=0x7f4154100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 --2- 172.21.15.133:0/3529884031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f414400b5f0 tx=0x7f4144006ee0).stop 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- 172.21.15.133:0/3529884031 shutdown_connections 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 --2- 172.21.15.133:0/3529884031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f4154100c20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- 172.21.15.133:0/3529884031 >> 172.21.15.133:0/3529884031 conn(0x7f41540f6280 msgr2=0x7f41540f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- 172.21.15.133:0/3529884031 shutdown_connections 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- 172.21.15.133:0/3529884031 wait complete. 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 Processor -- start 2023-11-16T01:07:52.627 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 -- start start 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.622+0000 7f4158f8a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4158f8a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415410d0c0 con 0x7f415406fc50 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f415259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f415259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54262/0 (socket says 172.21.15.133:54262) 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f415259c700 1 -- 172.21.15.133:0/570415472 learned_addr learned my addr 172.21.15.133:0/570415472 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f415259c700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414400b200 con 0x7f415406fc50 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f415259c700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f414400b1d0 tx=0x7f4144006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4144011690 con 0x7f415406fc50 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f414401b880 con 0x7f415406fc50 2023-11-16T01:07:52.628 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415410d2c0 con 0x7f415406fc50 2023-11-16T01:07:52.629 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414401a650 con 0x7f415406fc50 2023-11-16T01:07:52.629 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.623+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415410d6d0 con 0x7f415406fc50 2023-11-16T01:07:52.630 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f414400f070 con 0x7f415406fc50 2023-11-16T01:07:52.630 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4142ffd700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:52.630 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41380328d0 con 0x7f413802fd50 2023-11-16T01:07:52.631 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4144040ff0 con 0x7f415406fc50 2023-11-16T01:07:52.631 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4151d9b700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:52.631 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.624+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T01:07:52.826 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.825+0000 7f4151d9b700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:52.827 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:52.825+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T01:07:53.227 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:53.226+0000 7f4151d9b700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:53.228 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:53.226+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-16T01:07:54.029 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:54.028+0000 7f4151d9b700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:54.029 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:54.028+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-16T01:07:55.630 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:55.629+0000 7f4151d9b700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/4094884839 2023-11-16T01:07:55.630 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:55.629+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-16T01:07:55.948 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:55.947+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f414401a910 con 0x7f415406fc50 2023-11-16T01:07:55.948 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:55.947+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 msgr2=0x7f4138032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T01:07:55.948 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:55.947+0000 7f4142ffd700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:07:56.950 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.949+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f414401ac90 con 0x7f415406fc50 2023-11-16T01:07:56.952 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.949+0000 7f4142ffd700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 0x7f4138035c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:56.952 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.949+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41380328d0 con 0x7f4138033870 2023-11-16T01:07:56.953 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.950+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 0x7f4138035c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:56.953 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.950+0000 7f4151d9b700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 0x7f4138035c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4148006fd0 tx=0x7f4148006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:56.953 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.951+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f41380328d0 con 0x7f4138033870 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.957+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4154059100 con 0x7f4138033870 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4142ffd700 1 -- 172.21.15.133:0/570415472 <== mgr.14114 v2:172.21.15.133:6800/1494586672 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4154059100 con 0x7f4138033870 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "initialized": true 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 msgr2=0x7f4138035c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 0x7f4138035c60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4148006fd0 tx=0x7f4148006cf0).stop 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 msgr2=0x7f415410cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f414400b1d0 tx=0x7f4144006ee0).stop 2023-11-16T01:07:56.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 shutdown_connections 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f4138033870 0x7f4138035c60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f415406fc50 0x7f415410cbd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 --2- 172.21.15.133:0/570415472 >> [v2:172.21.15.133:6800/4094884839,v1:172.21.15.133:6801/4094884839] conn(0x7f413802fd50 0x7f4138032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 >> 172.21.15.133:0/570415472 conn(0x7f41540f6280 msgr2=0x7f41540f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 shutdown_connections 2023-11-16T01:07:56.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:56.958+0000 7f4158f8a700 1 -- 172.21.15.133:0/570415472 wait complete. 2023-11-16T01:07:57.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:55.895430+0000 mon.smithi133 (mon.0) 28 : cluster [INF] Active manager daemon smithi133.xdyttl restarted 2023-11-16T01:07:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:55.895979+0000 mon.smithi133 (mon.0) 29 : cluster [INF] Activating manager daemon smithi133.xdyttl 2023-11-16T01:07:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:55.897780+0000 mon.smithi133 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:55.948742+0000 mon.smithi133 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi133.xdyttl(active, starting, since 0.0528715s) 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.949777+0000 mon.smithi133 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.950409+0000 mon.smithi133 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mgr metadata", "who": "smithi133.xdyttl", "id": "smithi133.xdyttl"}]: dispatch 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.951076+0000 mon.smithi133 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.951238+0000 mon.smithi133 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T01:07:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.951377+0000 mon.smithi133 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:55.964712+0000 mon.smithi133 (mon.0) 37 : cluster [INF] Manager daemon smithi133.xdyttl is now available 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.980186+0000 mon.smithi133 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.985555+0000 mon.smithi133 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:55.990862+0000 mon.smithi133 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.080795+0000 mon.smithi133 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.081464+0000 mon.smithi133 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.083787+0000 mon.smithi133 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.085153+0000 mon.smithi133 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.094083+0000 mon.smithi133 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:07:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.095219+0000 mon.smithi133 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:07:57.323 INFO:teuthology.orchestra.run.smithi133.stderr:mgr epoch 4 is available 2023-11-16T01:07:57.324 INFO:teuthology.orchestra.run.smithi133.stderr:Setting orchestrator backend to cephadm... 2023-11-16T01:07:57.324 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.662+0000 7f84807b2700 1 Processor -- start 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.662+0000 7f84807b2700 1 -- start start 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.662+0000 7f84807b2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f84807b2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84780fed80 con 0x7f847806fc70 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847e54e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847e54e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56678/0 (socket says 172.21.15.133:56678) 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847e54e700 1 -- 172.21.15.133:0/2799762749 learned_addr learned my addr 172.21.15.133:0/2799762749 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847e54e700 1 -- 172.21.15.133:0/2799762749 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84780feec0 con 0x7f847806fc70 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847e54e700 1 --2- 172.21.15.133:0/2799762749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f846800b5f0 tx=0x7f8468006ee0).ready entity=mon.0 client_cookie=59fa1f910075b5ce server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847d54c700 1 -- 172.21.15.133:0/2799762749 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846800d600 con 0x7f847806fc70 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847d54c700 1 -- 172.21.15.133:0/2799762749 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f846800dbd0 con 0x7f847806fc70 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.663+0000 7f847d54c700 1 -- 172.21.15.133:0/2799762749 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8468011c00 con 0x7f847806fc70 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- 172.21.15.133:0/2799762749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 msgr2=0x7f84780fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 --2- 172.21.15.133:0/2799762749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f846800b5f0 tx=0x7f8468006ee0).stop 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- 172.21.15.133:0/2799762749 shutdown_connections 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 --2- 172.21.15.133:0/2799762749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fe800 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- 172.21.15.133:0/2799762749 >> 172.21.15.133:0/2799762749 conn(0x7f84780f6280 msgr2=0x7f84780f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- 172.21.15.133:0/2799762749 shutdown_connections 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- 172.21.15.133:0/2799762749 wait complete. 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 Processor -- start 2023-11-16T01:07:57.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.664+0000 7f84807b2700 1 -- start start 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f84807b2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f84807b2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84780fa700 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f847e54e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f847e54e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56690/0 (socket says 172.21.15.133:56690) 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f847e54e700 1 -- 172.21.15.133:0/1634194212 learned_addr learned my addr 172.21.15.133:0/1634194212 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f847e54e700 1 -- 172.21.15.133:0/1634194212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846800b200 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f847e54e700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f846800e040 tx=0x7f8468006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846801b410 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f846801b9e0 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84780fa8a0 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846801a770 con 0x7f847806fc70 2023-11-16T01:07:57.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.665+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84780facb0 con 0x7f847806fc70 2023-11-16T01:07:57.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.666+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f846800f070 con 0x7f847806fc70 2023-11-16T01:07:57.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.666+0000 7f8466ffd700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 0x7f84600321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:57.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.666+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8478045ad0 con 0x7f847806fc70 2023-11-16T01:07:57.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.666+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8468040f30 con 0x7f847806fc70 2023-11-16T01:07:57.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.666+0000 7f847dd4d700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 0x7f84600321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:57.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.667+0000 7f847dd4d700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 0x7f84600321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f84740096a0 tx=0x7f8474006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:57.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.671+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8468010750 con 0x7f847806fc70 2023-11-16T01:07:57.849 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.848+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f84780ffd60 con 0x7f846002fcf0 2023-11-16T01:07:57.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.853+0000 7f8466ffd700 1 -- 172.21.15.133:0/1634194212 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f84780ffd60 con 0x7f846002fcf0 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 msgr2=0x7f84600321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 0x7f84600321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f84740096a0 tx=0x7f8474006b40).stop 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 msgr2=0x7f84780fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f846800e040 tx=0x7f8468006ee0).stop 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 shutdown_connections 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f846002fcf0 0x7f84600321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 --2- 172.21.15.133:0/1634194212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f847806fc70 0x7f84780fc280 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 >> 172.21.15.133:0/1634194212 conn(0x7f84780f6280 msgr2=0x7f84780f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 shutdown_connections 2023-11-16T01:07:57.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:57.855+0000 7f84807b2700 1 -- 172.21.15.133:0/1634194212 wait complete. 2023-11-16T01:07:58.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.951226+0000 mgr.smithi133.xdyttl (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T01:07:58.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:56.952149+0000 mon.smithi133 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi133.xdyttl(active, since 1.05628s) 2023-11-16T01:07:58.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:56.958283+0000 mgr.smithi133.xdyttl (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T01:07:58.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:57.853717+0000 mon.smithi133 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:07:58.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:57.857161+0000 mon.smithi133 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:07:58.198 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-16T01:07:58.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.531+0000 7fe0c679f700 1 Processor -- start 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.531+0000 7fe0c679f700 1 -- start start 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.531+0000 7fe0c679f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.531+0000 7fe0c679f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00fed60 con 0x7fe0c006fc90 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0bffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0bffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56700/0 (socket says 172.21.15.133:56700) 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0bffff700 1 -- 172.21.15.133:0/2221616518 learned_addr learned my addr 172.21.15.133:0/2221616518 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0bffff700 1 -- 172.21.15.133:0/2221616518 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00feea0 con 0x7fe0c006fc90 2023-11-16T01:07:58.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0bffff700 1 --2- 172.21.15.133:0/2221616518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe0a800b5f0 tx=0x7fe0a8006ee0).ready entity=mon.0 client_cookie=5635cbfef2f1d3bd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0beffd700 1 -- 172.21.15.133:0/2221616518 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a800d600 con 0x7fe0c006fc90 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0beffd700 1 -- 172.21.15.133:0/2221616518 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe0a800dbd0 con 0x7fe0c006fc90 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.532+0000 7fe0beffd700 1 -- 172.21.15.133:0/2221616518 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a8011c00 con 0x7fe0c006fc90 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 -- 172.21.15.133:0/2221616518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 msgr2=0x7fe0c00fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 --2- 172.21.15.133:0/2221616518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe0a800b5f0 tx=0x7fe0a8006ee0).stop 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 -- 172.21.15.133:0/2221616518 shutdown_connections 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 --2- 172.21.15.133:0/2221616518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c00fe7e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 -- 172.21.15.133:0/2221616518 >> 172.21.15.133:0/2221616518 conn(0x7fe0c00f6280 msgr2=0x7fe0c00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 -- 172.21.15.133:0/2221616518 shutdown_connections 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 -- 172.21.15.133:0/2221616518 wait complete. 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.533+0000 7fe0c679f700 1 Processor -- start 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0c679f700 1 -- start start 2023-11-16T01:07:58.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0c679f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0c679f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c010aee0 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56716/0 (socket says 172.21.15.133:56716) 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bffff700 1 -- 172.21.15.133:0/4283428537 learned_addr learned my addr 172.21.15.133:0/4283428537 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bffff700 1 -- 172.21.15.133:0/4283428537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a800b200 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bffff700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe0a8000c00 tx=0x7fe0a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a8011690 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe0a800d9f0 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c010b0e0 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a801b710 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.534+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c010b4f0 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.535+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fe0a800f070 con 0x7fe0c006fc90 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.535+0000 7fe0bcff9700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 0x7fe0a0032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:58.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.535+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0045ad0 con 0x7fe0c006fc90 2023-11-16T01:07:58.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.535+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe0a80410f0 con 0x7fe0c006fc90 2023-11-16T01:07:58.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.536+0000 7fe0bf7fe700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 0x7fe0a0032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:58.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.536+0000 7fe0bf7fe700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 0x7fe0a0032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe0b00096a0 tx=0x7fe0b0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:58.540 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.540+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0a80226f0 con 0x7fe0c006fc90 2023-11-16T01:07:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.688+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0c010bdd0 con 0x7fe0a002fca0 2023-11-16T01:07:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.688+0000 7fe0bcff9700 1 -- 172.21.15.133:0/4283428537 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe0c010bdd0 con 0x7fe0a002fca0 2023-11-16T01:07:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: value unchanged 2023-11-16T01:07:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 msgr2=0x7fe0a0032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 0x7fe0a0032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe0b00096a0 tx=0x7fe0b0006b40).stop 2023-11-16T01:07:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 msgr2=0x7fe0c010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe0a8000c00 tx=0x7fe0a8006ee0).stop 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 shutdown_connections 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fe0a002fca0 0x7fe0a0032160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 --2- 172.21.15.133:0/4283428537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0c006fc90 0x7fe0c010a9f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 >> 172.21.15.133:0/4283428537 conn(0x7fe0c00f6280 msgr2=0x7fe0c00f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 shutdown_connections 2023-11-16T01:07:58.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:58.690+0000 7fe0c679f700 1 -- 172.21.15.133:0/4283428537 wait complete. 2023-11-16T01:07:58.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:58 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:57.849938+0000 mgr.smithi133.xdyttl (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:07:58.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:58 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:07:58.854510+0000 mon.smithi133 (mon.0) 50 : cluster [DBG] mgrmap e7: smithi133.xdyttl(active, since 2s) 2023-11-16T01:07:59.047 INFO:teuthology.orchestra.run.smithi133.stderr:Generating ssh key... 2023-11-16T01:07:59.047 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-16T01:07:59.389 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.384+0000 7fd5002cc700 1 Processor -- start 2023-11-16T01:07:59.389 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.384+0000 7fd5002cc700 1 -- start start 2023-11-16T01:07:59.389 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.384+0000 7fd5002cc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:59.389 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.384+0000 7fd5002cc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f80fdbe0 con 0x7fd4f80fd240 2023-11-16T01:07:59.389 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.384+0000 7fd4fe068700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fe068700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56732/0 (socket says 172.21.15.133:56732) 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fe068700 1 -- 172.21.15.133:0/737486529 learned_addr learned my addr 172.21.15.133:0/737486529 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fe068700 1 -- 172.21.15.133:0/737486529 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f80fdd20 con 0x7fd4f80fd240 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fe068700 1 --2- 172.21.15.133:0/737486529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd4e800b5f0 tx=0x7fd4e8006ee0).ready entity=mon.0 client_cookie=406f07bc710eb908 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fd066700 1 -- 172.21.15.133:0/737486529 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e800d600 con 0x7fd4f80fd240 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fd066700 1 -- 172.21.15.133:0/737486529 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4e800dbd0 con 0x7fd4f80fd240 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.385+0000 7fd4fd066700 1 -- 172.21.15.133:0/737486529 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8011c00 con 0x7fd4f80fd240 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- 172.21.15.133:0/737486529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 msgr2=0x7fd4f80fd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 --2- 172.21.15.133:0/737486529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd4e800b5f0 tx=0x7fd4e8006ee0).stop 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- 172.21.15.133:0/737486529 shutdown_connections 2023-11-16T01:07:59.390 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 --2- 172.21.15.133:0/737486529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fd660 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- 172.21.15.133:0/737486529 >> 172.21.15.133:0/737486529 conn(0x7fd4f80f59e0 msgr2=0x7fd4f80f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- 172.21.15.133:0/737486529 shutdown_connections 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- 172.21.15.133:0/737486529 wait complete. 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 Processor -- start 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.386+0000 7fd5002cc700 1 -- start start 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd5002cc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd5002cc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f80faf90 con 0x7fd4f80fd240 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4fe068700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4fe068700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56744/0 (socket says 172.21.15.133:56744) 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4fe068700 1 -- 172.21.15.133:0/2202162631 learned_addr learned my addr 172.21.15.133:0/2202162631 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4fe068700 1 -- 172.21.15.133:0/2202162631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e800b200 con 0x7fd4f80fd240 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4fe068700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd4e8002f40 tx=0x7fd4e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:59.391 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8011690 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4e801b880 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f80fb130 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e801a6b0 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.387+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f80fb540 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.388+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd4e800f070 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.388+0000 7fd4e6ffd700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 0x7fd4e0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.388+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd4e8041110 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.388+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8045ad0 con 0x7fd4f80fd240 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.388+0000 7fd4fd867700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 0x7fd4e0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:07:59.392 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.389+0000 7fd4fd867700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 0x7fd4e0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4f40096a0 tx=0x7fd4f4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:07:59.393 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.393+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4e8016070 con 0x7fd4f80fd240 2023-11-16T01:07:59.545 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.543+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4f80008d0 con 0x7fd4e002fdb0 2023-11-16T01:07:59.786 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.785+0000 7fd4e6ffd700 1 -- 172.21.15.133:0/2202162631 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd4f80008d0 con 0x7fd4e002fdb0 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 msgr2=0x7fd4e0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 0x7fd4e0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4f40096a0 tx=0x7fd4f4006b40).stop 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 msgr2=0x7fd4f80fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd4e8002f40 tx=0x7fd4e8006ee0).stop 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 shutdown_connections 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fd4e002fdb0 0x7fd4e0032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 --2- 172.21.15.133:0/2202162631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4f80fd240 0x7fd4f80fcb10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:07:59.788 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 >> 172.21.15.133:0/2202162631 conn(0x7fd4f80f59e0 msgr2=0x7fd4f80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:07:59.789 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 shutdown_connections 2023-11-16T01:07:59.789 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:07:59.787+0000 7fd5002cc700 1 -- 172.21.15.133:0/2202162631 wait complete. 2023-11-16T01:08:00.107 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:58.689103+0000 mgr.smithi133.xdyttl (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:00.108 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:59.783144+0000 mon.smithi133 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:00.108 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:07:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:07:59.784866+0000 mon.smithi133 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:00.131 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.499+0000 7f845c2c0700 1 Processor -- start 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.499+0000 7f845c2c0700 1 -- start start 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845c2c0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845c2c0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540fed20 con 0x7f845406fcd0 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845a05c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:00.503 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845a05c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56748/0 (socket says 172.21.15.133:56748) 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845a05c700 1 -- 172.21.15.133:0/2541624226 learned_addr learned my addr 172.21.15.133:0/2541624226 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845a05c700 1 -- 172.21.15.133:0/2541624226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84540fee60 con 0x7f845406fcd0 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845a05c700 1 --2- 172.21.15.133:0/2541624226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f844400b5f0 tx=0x7f8444006ee0).ready entity=mon.0 client_cookie=7ddbad1c4b176915 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845905a700 1 -- 172.21.15.133:0/2541624226 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f844400d600 con 0x7f845406fcd0 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845905a700 1 -- 172.21.15.133:0/2541624226 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f844400dbd0 con 0x7f845406fcd0 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.500+0000 7f845905a700 1 -- 172.21.15.133:0/2541624226 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8444011c00 con 0x7f845406fcd0 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- 172.21.15.133:0/2541624226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 msgr2=0x7f84540fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 --2- 172.21.15.133:0/2541624226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f844400b5f0 tx=0x7f8444006ee0).stop 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- 172.21.15.133:0/2541624226 shutdown_connections 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 --2- 172.21.15.133:0/2541624226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fe7a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- 172.21.15.133:0/2541624226 >> 172.21.15.133:0/2541624226 conn(0x7f84540f6280 msgr2=0x7f84540f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:00.504 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- 172.21.15.133:0/2541624226 shutdown_connections 2023-11-16T01:08:00.505 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- 172.21.15.133:0/2541624226 wait complete. 2023-11-16T01:08:00.505 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 Processor -- start 2023-11-16T01:08:00.505 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.501+0000 7f845c2c0700 1 -- start start 2023-11-16T01:08:00.505 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845c2c0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845c2c0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540fa6a0 con 0x7f845406fcd0 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845a05c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845a05c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56764/0 (socket says 172.21.15.133:56764) 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845a05c700 1 -- 172.21.15.133:0/1447879236 learned_addr learned my addr 172.21.15.133:0/1447879236 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845a05c700 1 -- 172.21.15.133:0/1447879236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844400b200 con 0x7f845406fcd0 2023-11-16T01:08:00.506 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845a05c700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f844400e040 tx=0x7f8444006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:00.507 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f844401b410 con 0x7f845406fcd0 2023-11-16T01:08:00.507 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84540fa840 con 0x7f845406fcd0 2023-11-16T01:08:00.507 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f844401b9e0 con 0x7f845406fcd0 2023-11-16T01:08:00.507 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f844401a770 con 0x7f845406fcd0 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.502+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84540fac50 con 0x7f845406fcd0 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.503+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8454045ad0 con 0x7f845406fcd0 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.503+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f844400f070 con 0x7f845406fcd0 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.503+0000 7f8442ffd700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 0x7f843c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.503+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8444041010 con 0x7f845406fcd0 2023-11-16T01:08:00.508 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.504+0000 7f845985b700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 0x7f843c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:00.509 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.504+0000 7f845985b700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 0x7f843c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f84500096a0 tx=0x7f8450006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:00.509 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.508+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8444019950 con 0x7f845406fcd0 2023-11-16T01:08:00.660 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.659+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8454104960 con 0x7f843c02fd50 2023-11-16T01:08:00.660 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.659+0000 7f8442ffd700 1 -- 172.21.15.133:0/1447879236 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f8454104960 con 0x7f843c02fd50 2023-11-16T01:08:00.660 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5aeFol3H9vaj+lHXXAdguOGz0IKq3pfRGGnwM6ADkPU1p/KR8BlsbhayYu42+dm5b6bZN2HlGLkpAuggkq69bCGpr24N7RS+jgwR/J0Tz1cX4DdMA57wraqWRQWL6OwKUHF2JlV4yUkA00z1i1XrUgW6JPJDvLOVbRowtwx58XbFjF9A96jrQ6YfgyV79LMQYno+B5sJBH4BPervxsT6h+9rPLcMr6nkwNRZC9qyTlZ9MkZ2Pi4rsOluP2vhlfVf4h84c8ygdmFdN3AjkG4UV3ruWL3DdohcaprUsbftfY6dLVyVeZtGNqyVTKut7hTsVdACpWITIxjRbR2SbsOXx03MQHxx1mBQYj+tamPcC48lITZ+aVh69Yn6UdSEoHWx6vHbc0QDMJJdt/1UeZwNO6n7h4X2OLT5ubKWfxZZ3ljeoz3k+gwTwYk4r88TRo2OahVt2KIdGks9itkzLGhqLVU1dI+GVkqni47AdOaS16bnmsnM7/+PdaAgUFw9WiM= ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 msgr2=0x7f843c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 0x7f843c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f84500096a0 tx=0x7f8450006b40).stop 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 msgr2=0x7f84540fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f844400e040 tx=0x7f8444006ee0).stop 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 shutdown_connections 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f843c02fd50 0x7f843c032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 --2- 172.21.15.133:0/1447879236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f845406fcd0 0x7f84540fc220 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:00.663 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 >> 172.21.15.133:0/1447879236 conn(0x7f84540f6280 msgr2=0x7f84540f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:00.664 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 shutdown_connections 2023-11-16T01:08:00.664 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:00.661+0000 7f845c2c0700 1 -- 172.21.15.133:0/1447879236 wait complete. 2023-11-16T01:08:01.014 INFO:teuthology.orchestra.run.smithi133.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T01:08:01.014 INFO:teuthology.orchestra.run.smithi133.stderr:Adding key to root@localhost authorized_keys... 2023-11-16T01:08:01.015 INFO:teuthology.orchestra.run.smithi133.stderr:Adding host smithi133... 2023-11-16T01:08:01.015 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi133 172.21.15.133 2023-11-16T01:08:01.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:07:59.544872+0000 mgr.smithi133.xdyttl (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.369+0000 7f71c2c32700 1 Processor -- start 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.369+0000 7f71c2c32700 1 -- start start 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.369+0000 7f71c2c32700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.369+0000 7f71c2c32700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc072090 con 0x7f71bc071c70 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71c09ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71c09ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56774/0 (socket says 172.21.15.133:56774) 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71c09ce700 1 -- 172.21.15.133:0/1718743801 learned_addr learned my addr 172.21.15.133:0/1718743801 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:01.372 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71c09ce700 1 -- 172.21.15.133:0/1718743801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc070700 con 0x7f71bc071c70 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71c09ce700 1 --2- 172.21.15.133:0/1718743801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f71b000b5f0 tx=0x7f71b0006ee0).ready entity=mon.0 client_cookie=c5209de1ac3b6260 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71bb7fe700 1 -- 172.21.15.133:0/1718743801 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b000d600 con 0x7f71bc071c70 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71bb7fe700 1 -- 172.21.15.133:0/1718743801 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f71b000dbd0 con 0x7f71bc071c70 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.370+0000 7f71bb7fe700 1 -- 172.21.15.133:0/1718743801 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b0011c00 con 0x7f71bc071c70 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 -- 172.21.15.133:0/1718743801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 msgr2=0x7f71bc070210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 --2- 172.21.15.133:0/1718743801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f71b000b5f0 tx=0x7f71b0006ee0).stop 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 -- 172.21.15.133:0/1718743801 shutdown_connections 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 --2- 172.21.15.133:0/1718743801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc070210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 -- 172.21.15.133:0/1718743801 >> 172.21.15.133:0/1718743801 conn(0x7f71bc0f73f0 msgr2=0x7f71bc0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 -- 172.21.15.133:0/1718743801 shutdown_connections 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 -- 172.21.15.133:0/1718743801 wait complete. 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.371+0000 7f71c2c32700 1 Processor -- start 2023-11-16T01:08:01.373 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c2c32700 1 -- start start 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c2c32700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c2c32700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc10d070 con 0x7f71bc071c70 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c09ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c09ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56784/0 (socket says 172.21.15.133:56784) 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c09ce700 1 -- 172.21.15.133:0/555135497 learned_addr learned my addr 172.21.15.133:0/555135497 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c09ce700 1 -- 172.21.15.133:0/555135497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b000b200 con 0x7f71bc071c70 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.372+0000 7f71c09ce700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f71b000b190 tx=0x7f71b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:01.377 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b001b410 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f71b001b9e0 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc10d270 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b001a880 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc10d680 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.373+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f71b000f070 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.374+0000 7f71b97fa700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 0x7f71a0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.374+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f71b00412f0 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.374+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc045ad0 con 0x7f71bc071c70 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.374+0000 7f71bbfff700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 0x7f71a0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:01.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.374+0000 7f71bbfff700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 0x7f71a0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71a80096a0 tx=0x7f71a8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:01.379 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.378+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b0016070 con 0x7f71bc071c70 2023-11-16T01:08:01.551 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:01.550+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi133", "addr": "172.21.15.133", "target": ["mon-mgr", ""]}) v1 -- 0x7f71bc106a70 con 0x7f71a002fd50 2023-11-16T01:08:02.235 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Added host 'smithi133' 2023-11-16T01:08:02.235 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.233+0000 7f71b97fa700 1 -- 172.21.15.133:0/555135497 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f71bc106a70 con 0x7f71a002fd50 2023-11-16T01:08:02.236 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 msgr2=0x7f71a0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:02.236 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 0x7f71a0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71a80096a0 tx=0x7f71a8006b40).stop 2023-11-16T01:08:02.237 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 msgr2=0x7f71bc10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:02.237 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f71b000b190 tx=0x7f71b0006ee0).stop 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 shutdown_connections 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f71a002fd50 0x7f71a0032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 --2- 172.21.15.133:0/555135497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f71bc071c70 0x7f71bc10cb80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 >> 172.21.15.133:0/555135497 conn(0x7f71bc0f73f0 msgr2=0x7f71bc0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 shutdown_connections 2023-11-16T01:08:02.238 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:02.236+0000 7f71c2c32700 1 -- 172.21.15.133:0/555135497 wait complete. 2023-11-16T01:08:02.268 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:00.659899+0000 mgr.smithi133.xdyttl (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:03.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:01.551422+0000 mgr.smithi133.xdyttl (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi133", "addr": "172.21.15.133", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:03.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:02.233464+0000 mon.smithi133 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:03.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:02.233948+0000 mon.smithi133 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:08:03.839 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying mon service with default placement... 2023-11-16T01:08:03.839 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-16T01:08:04.275 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:02.233742+0000 mgr.smithi133.xdyttl (mgr.14114) 9 : cephadm [INF] Added host smithi133 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.029+0000 7fa5438ef700 1 Processor -- start 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa5438ef700 1 -- start start 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa5438ef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa5438ef700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c0fb480 con 0x7fa53c0faae0 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa54168b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa54168b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57014/0 (socket says 172.21.15.133:57014) 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa54168b700 1 -- 172.21.15.133:0/3447734564 learned_addr learned my addr 172.21.15.133:0/3447734564 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:05.260 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.030+0000 7fa54168b700 1 -- 172.21.15.133:0/3447734564 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c0fb5c0 con 0x7fa53c0faae0 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa54168b700 1 --2- 172.21.15.133:0/3447734564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa52c00b5f0 tx=0x7fa52c006ee0).ready entity=mon.0 client_cookie=5beda15cac44a0c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa52bfff700 1 -- 172.21.15.133:0/3447734564 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c00d600 con 0x7fa53c0faae0 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa52bfff700 1 -- 172.21.15.133:0/3447734564 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa52c00dbd0 con 0x7fa53c0faae0 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa52bfff700 1 -- 172.21.15.133:0/3447734564 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c011c00 con 0x7fa53c0faae0 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 -- 172.21.15.133:0/3447734564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 msgr2=0x7fa53c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3447734564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa52c00b5f0 tx=0x7fa52c006ee0).stop 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 -- 172.21.15.133:0/3447734564 shutdown_connections 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3447734564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c0faf00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 -- 172.21.15.133:0/3447734564 >> 172.21.15.133:0/3447734564 conn(0x7fa53c0f6280 msgr2=0x7fa53c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 -- 172.21.15.133:0/3447734564 shutdown_connections 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.031+0000 7fa5438ef700 1 -- 172.21.15.133:0/3447734564 wait complete. 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa5438ef700 1 Processor -- start 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa5438ef700 1 -- start start 2023-11-16T01:08:05.261 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa5438ef700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa5438ef700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c109dc0 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa54168b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa54168b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57016/0 (socket says 172.21.15.133:57016) 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa54168b700 1 -- 172.21.15.133:0/3736820352 learned_addr learned my addr 172.21.15.133:0/3736820352 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa54168b700 1 -- 172.21.15.133:0/3736820352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c00b200 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.032+0000 7fa54168b700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa52c00b1d0 tx=0x7fa52c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.033+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c011a20 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.033+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa52c01b880 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.033+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c109fc0 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.033+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c01a700 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.033+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c10a3d0 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.034+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa52c00f070 con 0x7fa53c0faae0 2023-11-16T01:08:05.262 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.034+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c045ad0 con 0x7fa53c0faae0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.034+0000 7fa529ffb700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 0x7fa524032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.034+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa52c041090 con 0x7fa53c0faae0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.034+0000 7fa540e8a700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 0x7fa524032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.035+0000 7fa540e8a700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 0x7fa524032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa5380096a0 tx=0x7fa538006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.038+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa52c019cd0 con 0x7fa53c0faae0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.215+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa53c10ad10 con 0x7fa52402fdb0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.219+0000 7fa529ffb700 1 -- 172.21.15.133:0/3736820352 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa53c10ad10 con 0x7fa52402fdb0 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 msgr2=0x7fa524032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 0x7fa524032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa5380096a0 tx=0x7fa538006b40).stop 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 msgr2=0x7fa53c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa52c00b1d0 tx=0x7fa52c006ee0).stop 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 shutdown_connections 2023-11-16T01:08:05.263 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7fa52402fdb0 0x7fa524032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:05.264 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 --2- 172.21.15.133:0/3736820352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa53c0faae0 0x7fa53c1098d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:05.264 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 >> 172.21.15.133:0/3736820352 conn(0x7fa53c0f6280 msgr2=0x7fa53c0f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:05.264 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 shutdown_connections 2023-11-16T01:08:05.264 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:05.221+0000 7fa5438ef700 1 -- 172.21.15.133:0/3736820352 wait complete. 2023-11-16T01:08:05.264 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-16T01:08:06.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:05.216390+0000 mgr.smithi133.xdyttl (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:06.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:05.217079+0000 mgr.smithi133.xdyttl (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-16T01:08:06.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:05.219347+0000 mon.smithi133 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:07.496 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying mgr service with default placement... 2023-11-16T01:08:07.496 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f113b157700 1 Processor -- start 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f113b157700 1 -- start start 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f113b157700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f113b157700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340fed60 con 0x7f113406fc90 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f1138ef3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:09.058 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f1138ef3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57032/0 (socket says 172.21.15.133:57032) 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.815+0000 7f1138ef3700 1 -- 172.21.15.133:0/792470749 learned_addr learned my addr 172.21.15.133:0/792470749 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f1138ef3700 1 -- 172.21.15.133:0/792470749 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11340feea0 con 0x7f113406fc90 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f1138ef3700 1 --2- 172.21.15.133:0/792470749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f112800b5f0 tx=0x7f1128006ee0).ready entity=mon.0 client_cookie=2f95b7f45e69fe49 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f11337fe700 1 -- 172.21.15.133:0/792470749 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112800d600 con 0x7f113406fc90 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f11337fe700 1 -- 172.21.15.133:0/792470749 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f112800dbd0 con 0x7f113406fc90 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f11337fe700 1 -- 172.21.15.133:0/792470749 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1128011c00 con 0x7f113406fc90 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f113b157700 1 -- 172.21.15.133:0/792470749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 msgr2=0x7f11340fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f113b157700 1 --2- 172.21.15.133:0/792470749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f112800b5f0 tx=0x7f1128006ee0).stop 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.816+0000 7f113b157700 1 -- 172.21.15.133:0/792470749 shutdown_connections 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 --2- 172.21.15.133:0/792470749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fe7e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 -- 172.21.15.133:0/792470749 >> 172.21.15.133:0/792470749 conn(0x7f11340f6280 msgr2=0x7f11340f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:09.059 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 -- 172.21.15.133:0/792470749 shutdown_connections 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 -- 172.21.15.133:0/792470749 wait complete. 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 Processor -- start 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.817+0000 7f113b157700 1 -- start start 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f113b157700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f113b157700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340fa6e0 con 0x7f113406fc90 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f1138ef3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f1138ef3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57044/0 (socket says 172.21.15.133:57044) 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f1138ef3700 1 -- 172.21.15.133:0/2254449381 learned_addr learned my addr 172.21.15.133:0/2254449381 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f1138ef3700 1 -- 172.21.15.133:0/2254449381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112800b200 con 0x7f113406fc90 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f1138ef3700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f112800e040 tx=0x7f1128006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112801b410 con 0x7f113406fc90 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f112801b9e0 con 0x7f113406fc90 2023-11-16T01:08:09.060 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11340fa880 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f112801a770 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.818+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11340fac90 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.819+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f112800f070 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.819+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1134045ad0 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.820+0000 7f11317fa700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 0x7f1118032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.820+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1128041030 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.820+0000 7f1133fff700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 0x7f1118032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.820+0000 7f1133fff700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 0x7f1118032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f11200096a0 tx=0x7f1120006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:08.824+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1128019950 con 0x7f113406fc90 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.004+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1134104960 con 0x7f111802fd50 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.008+0000 7f11317fa700 1 -- 172.21.15.133:0/2254449381 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1134104960 con 0x7f111802fd50 2023-11-16T01:08:09.061 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 msgr2=0x7f1118032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 0x7f1118032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f11200096a0 tx=0x7f1120006b40).stop 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 msgr2=0x7f11340fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f112800e040 tx=0x7f1128006ee0).stop 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 shutdown_connections 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f111802fd50 0x7f1118032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 --2- 172.21.15.133:0/2254449381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f113406fc90 0x7f11340fc260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 >> 172.21.15.133:0/2254449381 conn(0x7f11340f6280 msgr2=0x7f11340f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 shutdown_connections 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:09.009+0000 7f113b157700 1 -- 172.21.15.133:0/2254449381 wait complete. 2023-11-16T01:08:09.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-16T01:08:10.171 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying crash service with default placement... 2023-11-16T01:08:10.171 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-16T01:08:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:09.005224+0000 mgr.smithi133.xdyttl (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:09.005758+0000 mgr.smithi133.xdyttl (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-16T01:08:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:09.008039+0000 mon.smithi133 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.635+0000 7f165c7c4700 1 Processor -- start 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165c7c4700 1 -- start start 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165c7c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165c7c4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540fed80 con 0x7f165406fc70 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165a560700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165a560700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57052/0 (socket says 172.21.15.133:57052) 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165a560700 1 -- 172.21.15.133:0/3304516364 learned_addr learned my addr 172.21.15.133:0/3304516364 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.636+0000 7f165a560700 1 -- 172.21.15.133:0/3304516364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540feec0 con 0x7f165406fc70 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165a560700 1 --2- 172.21.15.133:0/3304516364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f165000b5f0 tx=0x7f1650006ee0).ready entity=mon.0 client_cookie=26a42c97ae47153a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165955e700 1 -- 172.21.15.133:0/3304516364 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f165000d600 con 0x7f165406fc70 2023-11-16T01:08:10.639 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165955e700 1 -- 172.21.15.133:0/3304516364 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f165000dbd0 con 0x7f165406fc70 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165955e700 1 -- 172.21.15.133:0/3304516364 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1650011c00 con 0x7f165406fc70 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165c7c4700 1 -- 172.21.15.133:0/3304516364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 msgr2=0x7f16540fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3304516364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f165000b5f0 tx=0x7f1650006ee0).stop 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165c7c4700 1 -- 172.21.15.133:0/3304516364 shutdown_connections 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3304516364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fe800 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.637+0000 7f165c7c4700 1 -- 172.21.15.133:0/3304516364 >> 172.21.15.133:0/3304516364 conn(0x7f16540f6280 msgr2=0x7f16540f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.638+0000 7f165c7c4700 1 -- 172.21.15.133:0/3304516364 shutdown_connections 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.638+0000 7f165c7c4700 1 -- 172.21.15.133:0/3304516364 wait complete. 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.638+0000 7f165c7c4700 1 Processor -- start 2023-11-16T01:08:10.640 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.638+0000 7f165c7c4700 1 -- start start 2023-11-16T01:08:10.642 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165c7c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:10.642 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165c7c4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540fa700 con 0x7f165406fc70 2023-11-16T01:08:10.642 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165a560700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165a560700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57062/0 (socket says 172.21.15.133:57062) 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165a560700 1 -- 172.21.15.133:0/3403546555 learned_addr learned my addr 172.21.15.133:0/3403546555 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165a560700 1 -- 172.21.15.133:0/3403546555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165000b200 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.639+0000 7f165a560700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1650000c00 tx=0x7f1650006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1650011690 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16540fa8a0 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f165000d9f0 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f165001b710 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16540facb0 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.640+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f165000f070 con 0x7f165406fc70 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.641+0000 7f1642ffd700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 0x7f163c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:10.643 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.641+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1650041160 con 0x7f165406fc70 2023-11-16T01:08:10.644 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.641+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654045ad0 con 0x7f165406fc70 2023-11-16T01:08:10.644 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.641+0000 7f1659d5f700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 0x7f163c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:10.644 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.641+0000 7f1659d5f700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 0x7f163c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f16440096a0 tx=0x7f1644006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:10.646 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.645+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1650016070 con 0x7f165406fc70 2023-11-16T01:08:10.841 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.839+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f16540008d0 con 0x7f163c02fd50 2023-11-16T01:08:10.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.843+0000 7f1642ffd700 1 -- 172.21.15.133:0/3403546555 <== mgr.14114 v2:172.21.15.133:6800/1494586672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f16540008d0 con 0x7f163c02fd50 2023-11-16T01:08:10.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-16T01:08:10.846 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 msgr2=0x7f163c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:10.846 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 0x7f163c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f16440096a0 tx=0x7f1644006b40).stop 2023-11-16T01:08:10.846 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 msgr2=0x7f16540fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1650000c00 tx=0x7f1650006ee0).stop 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 shutdown_connections 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f163c02fd50 0x7f163c032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 --2- 172.21.15.133:0/3403546555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f165406fc70 0x7f16540fc280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 >> 172.21.15.133:0/3403546555 conn(0x7f16540f6280 msgr2=0x7f16540f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.845+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 shutdown_connections 2023-11-16T01:08:10.847 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:10.846+0000 7f165c7c4700 1 -- 172.21.15.133:0/3403546555 wait complete. 2023-11-16T01:08:11.311 INFO:teuthology.orchestra.run.smithi133.stderr:Enabling mgr prometheus module... 2023-11-16T01:08:11.343 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.664+0000 7f1d87508700 1 Processor -- start 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.664+0000 7f1d87508700 1 -- start start 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d87508700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d87508700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800fb480 con 0x7f1d800faae0 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d852a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d852a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57068/0 (socket says 172.21.15.133:57068) 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d852a4700 1 -- 172.21.15.133:0/218244862 learned_addr learned my addr 172.21.15.133:0/218244862 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.665+0000 7f1d852a4700 1 -- 172.21.15.133:0/218244862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800fb5c0 con 0x7f1d800faae0 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d852a4700 1 --2- 172.21.15.133:0/218244862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1d7c00b5f0 tx=0x7f1d7c006ee0).ready entity=mon.0 client_cookie=75c45cac06308ec3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d6ffff700 1 -- 172.21.15.133:0/218244862 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c00d600 con 0x7f1d800faae0 2023-11-16T01:08:11.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d6ffff700 1 -- 172.21.15.133:0/218244862 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d7c00dc10 con 0x7f1d800faae0 2023-11-16T01:08:11.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d6ffff700 1 -- 172.21.15.133:0/218244862 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c011cd0 con 0x7f1d800faae0 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d87508700 1 -- 172.21.15.133:0/218244862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 msgr2=0x7f1d800faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.666+0000 7f1d87508700 1 --2- 172.21.15.133:0/218244862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1d7c00b5f0 tx=0x7f1d7c006ee0).stop 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.667+0000 7f1d87508700 1 -- 172.21.15.133:0/218244862 shutdown_connections 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.667+0000 7f1d87508700 1 --2- 172.21.15.133:0/218244862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d800faf00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.667+0000 7f1d87508700 1 -- 172.21.15.133:0/218244862 >> 172.21.15.133:0/218244862 conn(0x7f1d800f6280 msgr2=0x7f1d800f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.667+0000 7f1d87508700 1 -- 172.21.15.133:0/218244862 shutdown_connections 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.667+0000 7f1d87508700 1 -- 172.21.15.133:0/218244862 wait complete. 2023-11-16T01:08:11.670 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d87508700 1 Processor -- start 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d87508700 1 -- start start 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d87508700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d87508700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8010d740 con 0x7f1d800faae0 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d852a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d852a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57082/0 (socket says 172.21.15.133:57082) 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d852a4700 1 -- 172.21.15.133:0/1607174157 learned_addr learned my addr 172.21.15.133:0/1607174157 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d852a4700 1 -- 172.21.15.133:0/1607174157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c00b200 con 0x7f1d800faae0 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.668+0000 7f1d852a4700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1d7c006800 tx=0x7f1d7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:11.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c011df0 con 0x7f1d800faae0 2023-11-16T01:08:11.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d7c01b8c0 con 0x7f1d800faae0 2023-11-16T01:08:11.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8010d880 con 0x7f1d800faae0 2023-11-16T01:08:11.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c01a790 con 0x7f1d800faae0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8010dcf0 con 0x7f1d800faae0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.669+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1d7c00f070 con 0x7f1d800faae0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.670+0000 7f1d6dffb700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 0x7f1d68032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.670+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1d7c041d20 con 0x7f1d800faae0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.670+0000 7f1d84aa3700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 0x7f1d68032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.670+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d80045ad0 con 0x7f1d800faae0 2023-11-16T01:08:11.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.670+0000 7f1d84aa3700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 0x7f1d68032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1d700096a0 tx=0x7f1d70006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:11.675 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.674+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d7c024410 con 0x7f1d800faae0 2023-11-16T01:08:11.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:11.875+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f1d80027a00 con 0x7f1d800faae0 2023-11-16T01:08:12.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:10.840895+0000 mgr.smithi133.xdyttl (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:12.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:10.841559+0000 mgr.smithi133.xdyttl (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-16T01:08:12.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:10.843692+0000 mon.smithi133 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:12.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:11.092838+0000 mon.smithi133 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:12.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:11.368579+0000 mon.smithi133 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:12.848 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.846+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f1d7c024790 con 0x7f1d800faae0 2023-11-16T01:08:12.849 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.847+0000 7f1d6dffb700 1 -- 172.21.15.133:0/1607174157 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f1d7c0194e0 con 0x7f1d800faae0 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 msgr2=0x7f1d68032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 0x7f1d68032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1d700096a0 tx=0x7f1d70006b40).stop 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 msgr2=0x7f1d8006fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1d7c006800 tx=0x7f1d7c006ee0).stop 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 shutdown_connections 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.849+0000 7f1d87508700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7f1d6802fd50 0x7f1d68032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.850+0000 7f1d87508700 1 --2- 172.21.15.133:0/1607174157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d800faae0 0x7f1d8006fbe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.850+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 >> 172.21.15.133:0/1607174157 conn(0x7f1d800f6280 msgr2=0x7f1d800f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.850+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 shutdown_connections 2023-11-16T01:08:12.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:12.850+0000 7f1d87508700 1 -- 172.21.15.133:0/1607174157 wait complete. 2023-11-16T01:08:13.194 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:12 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:11.877126+0000 mon.smithi133 (mon.0) 60 : audit [INF] from='client.? 172.21.15.133:0/1607174157' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-16T01:08:13.194 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:12 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:12.231980+0000 mon.smithi133 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.133:0/2321859253' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:13.396 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying prometheus service with default placement... 2023-11-16T01:08:13.396 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-16T01:08:13.774 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff32a8d9700 1 Processor -- start 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff32a8d9700 1 -- start start 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff32a8d9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff32a8d9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240fc4e0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff323fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff323fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53700/0 (socket says 172.21.15.133:53700) 2023-11-16T01:08:13.775 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff323fff700 1 -- 172.21.15.133:0/2438489893 learned_addr learned my addr 172.21.15.133:0/2438489893 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.769+0000 7ff323fff700 1 -- 172.21.15.133:0/2438489893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240fc620 con 0x7ff3240fbbd0 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff323fff700 1 --2- 172.21.15.133:0/2438489893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=69872c3dacb7f6c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff322ffd700 1 -- 172.21.15.133:0/2438489893 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31400d600 con 0x7ff3240fbbd0 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff322ffd700 1 -- 172.21.15.133:0/2438489893 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff31400dc10 con 0x7ff3240fbbd0 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff322ffd700 1 -- 172.21.15.133:0/2438489893 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff314011cd0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2438489893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 msgr2=0x7ff3240fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.770+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2438489893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).stop 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2438489893 shutdown_connections 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2438489893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff3240fbff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:13.776 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2438489893 >> 172.21.15.133:0/2438489893 conn(0x7ff3240f73f0 msgr2=0x7ff3240f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2438489893 shutdown_connections 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2438489893 wait complete. 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.771+0000 7ff32a8d9700 1 Processor -- start 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff32a8d9700 1 -- start start 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff32a8d9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff32a8d9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32410aef0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff323fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff323fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53714/0 (socket says 172.21.15.133:53714) 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff323fff700 1 -- 172.21.15.133:0/2475628619 learned_addr learned my addr 172.21.15.133:0/2475628619 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:13.777 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff323fff700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31400b200 con 0x7ff3240fbbd0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff323fff700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff31400dfb0 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.772+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff314011df0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.773+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff31401b8c0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.773+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32410b0f0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.773+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31401a8c0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.778 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.773+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32410b500 con 0x7ff3240fbbd0 2023-11-16T01:08:13.779 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.773+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7ff31400f070 con 0x7ff3240fbbd0 2023-11-16T01:08:13.779 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.774+0000 7ff320ff9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:13.779 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.774+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff314042910 con 0x7ff3240fbbd0 2023-11-16T01:08:13.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.774+0000 7ff3237fe700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1494586672 2023-11-16T01:08:13.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.774+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T01:08:13.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.774+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff324045ad0 con 0x7ff3240fbbd0 2023-11-16T01:08:13.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.778+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff314016030 con 0x7ff3240fbbd0 2023-11-16T01:08:13.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.961+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3241048c0 con 0x7ff31002fdb0 2023-11-16T01:08:13.976 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.975+0000 7ff3237fe700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1494586672 2023-11-16T01:08:13.977 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:13.975+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T01:08:14.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:12.849017+0000 mon.smithi133 (mon.0) 62 : audit [INF] from='client.? 172.21.15.133:0/1607174157' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-16T01:08:14.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:12.849075+0000 mon.smithi133 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi133.xdyttl(active, since 16s) 2023-11-16T01:08:14.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:14.376+0000 7ff3237fe700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1494586672 2023-11-16T01:08:14.378 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:14.376+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-16T01:08:15.178 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:15.177+0000 7ff3237fe700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1494586672 2023-11-16T01:08:15.179 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:15.177+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-16T01:08:16.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:16.778+0000 7ff3237fe700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1494586672 2023-11-16T01:08:16.780 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:16.778+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-16T01:08:17.823 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:17.820+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7ff314024930 con 0x7ff3240fbbd0 2023-11-16T01:08:17.823 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:17.821+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 msgr2=0x7ff310032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T01:08:17.823 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:17.821+0000 7ff320ff9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:17.770101+0000 mon.smithi133 (mon.0) 64 : cluster [INF] Active manager daemon smithi133.xdyttl restarted 2023-11-16T01:08:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:17.770513+0000 mon.smithi133 (mon.0) 65 : cluster [INF] Activating manager daemon smithi133.xdyttl 2023-11-16T01:08:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:17.772199+0000 mon.smithi133 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-16T01:08:18.828 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.825+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7ff314024410 con 0x7ff3240fbbd0 2023-11-16T01:08:18.829 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.826+0000 7ff320ff9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 0x7ff310036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:18.829 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.826+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 --> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3241048c0 con 0x7ff310033c20 2023-11-16T01:08:18.829 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.826+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 0x7ff310036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:18.829 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.827+0000 7ff3237fe700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 0x7ff310036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff3180036d0 tx=0x7ff318006c60).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:18.839 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.837+0000 7ff320ff9700 1 -- 172.21.15.133:0/2475628619 <== mgr.14142 v2:172.21.15.133:6800/3633665222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7ff3241048c0 con 0x7ff310033c20 2023-11-16T01:08:18.839 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-16T01:08:18.841 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 msgr2=0x7ff310036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 0x7ff310036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff3180036d0 tx=0x7ff318006c60).stop 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 msgr2=0x7ff32410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff31400dfb0 tx=0x7ff314006ee0).stop 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 shutdown_connections 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff310033c20 0x7ff310036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3240fbbd0 0x7ff32410aa00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 --2- 172.21.15.133:0/2475628619 >> [v2:172.21.15.133:6800/1494586672,v1:172.21.15.133:6801/1494586672] conn(0x7ff31002fdb0 0x7ff310032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 >> 172.21.15.133:0/2475628619 conn(0x7ff3240f73f0 msgr2=0x7ff3240f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 shutdown_connections 2023-11-16T01:08:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:18.839+0000 7ff32a8d9700 1 -- 172.21.15.133:0/2475628619 wait complete. 2023-11-16T01:08:19.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:17.823284+0000 mon.smithi133 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi133.xdyttl(active, starting, since 0.0528573s) 2023-11-16T01:08:19.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.824264+0000 mon.smithi133 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2023-11-16T01:08:19.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.826188+0000 mon.smithi133 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mgr metadata", "who": "smithi133.xdyttl", "id": "smithi133.xdyttl"}]: dispatch 2023-11-16T01:08:19.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.826409+0000 mon.smithi133 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.826538+0000 mon.smithi133 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.826665+0000 mon.smithi133 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:17.841006+0000 mon.smithi133 (mon.0) 73 : cluster [INF] Manager daemon smithi133.xdyttl is now available 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.940736+0000 mon.smithi133 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.942157+0000 mon.smithi133 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.956551+0000 mon.smithi133 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:17.959273+0000 mon.smithi133 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:08:19.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:18.805902+0000 mon.smithi133 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:19.293 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying grafana service with default placement... 2023-11-16T01:08:19.293 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-16T01:08:19.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.953+0000 7ff6a601f700 1 Processor -- start 2023-11-16T01:08:19.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.953+0000 7ff6a601f700 1 -- start start 2023-11-16T01:08:19.957 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff6a601f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff6a601f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a00fed20 con 0x7ff6a006fcd0 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53766/0 (socket says 172.21.15.133:53766) 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69f7fe700 1 -- 172.21.15.133:0/3921180379 learned_addr learned my addr 172.21.15.133:0/3921180379 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69f7fe700 1 -- 172.21.15.133:0/3921180379 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a00fee60 con 0x7ff6a006fcd0 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69f7fe700 1 --2- 172.21.15.133:0/3921180379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff68800b5f0 tx=0x7ff688006ee0).ready entity=mon.0 client_cookie=c4dc754c37c32663 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:19.958 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69e7fc700 1 -- 172.21.15.133:0/3921180379 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff68800d600 con 0x7ff6a006fcd0 2023-11-16T01:08:19.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69e7fc700 1 -- 172.21.15.133:0/3921180379 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff68800dc10 con 0x7ff6a006fcd0 2023-11-16T01:08:19.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.954+0000 7ff69e7fc700 1 -- 172.21.15.133:0/3921180379 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff688011cd0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.960 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 -- 172.21.15.133:0/3921180379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 msgr2=0x7ff6a00fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:19.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 --2- 172.21.15.133:0/3921180379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff68800b5f0 tx=0x7ff688006ee0).stop 2023-11-16T01:08:19.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 -- 172.21.15.133:0/3921180379 shutdown_connections 2023-11-16T01:08:19.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 --2- 172.21.15.133:0/3921180379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fe7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:19.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 -- 172.21.15.133:0/3921180379 >> 172.21.15.133:0/3921180379 conn(0x7ff6a00f6280 msgr2=0x7ff6a00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:19.961 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 -- 172.21.15.133:0/3921180379 shutdown_connections 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.955+0000 7ff6a601f700 1 -- 172.21.15.133:0/3921180379 wait complete. 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff6a601f700 1 Processor -- start 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff6a601f700 1 -- start start 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff6a601f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff6a601f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a00fa6a0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff69f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff69f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53776/0 (socket says 172.21.15.133:53776) 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff69f7fe700 1 -- 172.21.15.133:0/2941838384 learned_addr learned my addr 172.21.15.133:0/2941838384 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:19.962 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff69f7fe700 1 -- 172.21.15.133:0/2941838384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68800b200 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.956+0000 7ff69f7fe700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff68800b1d0 tx=0x7ff688006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff688011df0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a00fa840 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff68801b8c0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff68801a7b0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a00fac50 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.957+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff68800f070 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.958+0000 7ff697fff700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 0x7ff680032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.958+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff688042390 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.958+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a0045ad0 con 0x7ff6a006fcd0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.958+0000 7ff69effd700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 0x7ff680032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:19.963 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.958+0000 7ff69effd700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 0x7ff680032320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff6900096a0 tx=0x7ff690006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:19.964 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:19.962+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff688016070 con 0x7ff6a006fcd0 2023-11-16T01:08:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:18.828944+0000 mon.smithi133 (mon.0) 79 : cluster [DBG] mgrmap e10: smithi133.xdyttl(active, since 1.05852s) 2023-11-16T01:08:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:18.829999+0000 mgr.smithi133.xdyttl (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-16T01:08:20.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:18.838068+0000 mon.smithi133 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:20.139 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.137+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 --> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6a01048d0 con 0x7ff68002fe60 2023-11-16T01:08:20.143 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.141+0000 7ff697fff700 1 -- 172.21.15.133:0/2941838384 <== mgr.14142 v2:172.21.15.133:6800/3633665222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff6a01048d0 con 0x7ff68002fe60 2023-11-16T01:08:20.143 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-16T01:08:20.145 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 msgr2=0x7ff680032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 0x7ff680032320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff6900096a0 tx=0x7ff690006b40).stop 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 msgr2=0x7ff6a00fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff68800b1d0 tx=0x7ff688006ee0).stop 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 shutdown_connections 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff68002fe60 0x7ff680032320 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 --2- 172.21.15.133:0/2941838384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6a006fcd0 0x7ff6a00fc220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 >> 172.21.15.133:0/2941838384 conn(0x7ff6a00f6280 msgr2=0x7ff6a00f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.143+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 shutdown_connections 2023-11-16T01:08:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.144+0000 7ff6a601f700 1 -- 172.21.15.133:0/2941838384 wait complete. 2023-11-16T01:08:20.502 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying node-exporter service with default placement... 2023-11-16T01:08:20.502 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.898+0000 7fb7b99bd700 1 Processor -- start 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.898+0000 7fb7b99bd700 1 -- start start 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.898+0000 7fb7b99bd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.898+0000 7fb7b99bd700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b40fcad0 con 0x7fb7b40fc130 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53790/0 (socket says 172.21.15.133:53790) 2023-11-16T01:08:20.903 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b2ffd700 1 -- 172.21.15.133:0/2728821184 learned_addr learned my addr 172.21.15.133:0/2728821184 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b2ffd700 1 -- 172.21.15.133:0/2728821184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b40fcc10 con 0x7fb7b40fc130 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b2ffd700 1 --2- 172.21.15.133:0/2728821184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb7a400b5f0 tx=0x7fb7a4006ee0).ready entity=mon.0 client_cookie=67e90232457db6e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b1ffb700 1 -- 172.21.15.133:0/2728821184 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7a400d600 con 0x7fb7b40fc130 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b1ffb700 1 -- 172.21.15.133:0/2728821184 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7a400dc10 con 0x7fb7b40fc130 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.899+0000 7fb7b1ffb700 1 -- 172.21.15.133:0/2728821184 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7a4011cd0 con 0x7fb7b40fc130 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 -- 172.21.15.133:0/2728821184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 msgr2=0x7fb7b40fc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/2728821184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb7a400b5f0 tx=0x7fb7a4006ee0).stop 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 -- 172.21.15.133:0/2728821184 shutdown_connections 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/2728821184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b40fc550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 -- 172.21.15.133:0/2728821184 >> 172.21.15.133:0/2728821184 conn(0x7fb7b40f59e0 msgr2=0x7fb7b40f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 -- 172.21.15.133:0/2728821184 shutdown_connections 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 -- 172.21.15.133:0/2728821184 wait complete. 2023-11-16T01:08:20.904 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.900+0000 7fb7b99bd700 1 Processor -- start 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b99bd700 1 -- start start 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b99bd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b99bd700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b410d220 con 0x7fb7b40fc130 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53802/0 (socket says 172.21.15.133:53802) 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b2ffd700 1 -- 172.21.15.133:0/3564210387 learned_addr learned my addr 172.21.15.133:0/3564210387 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b2ffd700 1 -- 172.21.15.133:0/3564210387 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a400b200 con 0x7fb7b40fc130 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b2ffd700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb7a400dfe0 tx=0x7fb7a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:20.905 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7a4011df0 con 0x7fb7b40fc130 2023-11-16T01:08:20.906 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b410d420 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7a401b8c0 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7a401a840 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.901+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b410d830 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.902+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb7a400f070 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.902+0000 7fb7ab7fe700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 0x7fb794032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.902+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb7a401ba30 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.902+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b4045ad0 con 0x7fb7b40fc130 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.902+0000 7fb7b27fc700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 0x7fb794032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:20.907 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.903+0000 7fb7b27fc700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 0x7fb794032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb79c0096a0 tx=0x7fb79c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:20.908 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:20.907+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7a4016070 con 0x7fb7b40fc130 2023-11-16T01:08:21.086 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.084+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 --> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7b410df60 con 0x7fb79402fe60 2023-11-16T01:08:21.090 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.088+0000 7fb7ab7fe700 1 -- 172.21.15.133:0/3564210387 <== mgr.14142 v2:172.21.15.133:6800/3633665222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb7b410df60 con 0x7fb79402fe60 2023-11-16T01:08:21.090 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-16T01:08:21.093 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 msgr2=0x7fb794032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:21.093 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 0x7fb794032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb79c0096a0 tx=0x7fb79c006b40).stop 2023-11-16T01:08:21.093 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 msgr2=0x7fb7b410cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:21.093 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb7a400dfe0 tx=0x7fb7a4006ee0).stop 2023-11-16T01:08:21.093 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 shutdown_connections 2023-11-16T01:08:21.094 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb79402fe60 0x7fb794032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:21.094 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 --2- 172.21.15.133:0/3564210387 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7b40fc130 0x7fb7b410cd30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:21.094 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 >> 172.21.15.133:0/3564210387 conn(0x7fb7b40f59e0 msgr2=0x7fb7b40f7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:21.094 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 shutdown_connections 2023-11-16T01:08:21.094 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.090+0000 7fb7b99bd700 1 -- 172.21.15.133:0/3564210387 wait complete. 2023-11-16T01:08:21.114 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:19.841923+0000 mon.smithi133 (mon.0) 81 : cluster [DBG] mgrmap e11: smithi133.xdyttl(active, since 2s) 2023-11-16T01:08:21.114 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:20.142780+0000 mon.smithi133 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:21.494 INFO:teuthology.orchestra.run.smithi133.stderr:Deploying alertmanager service with default placement... 2023-11-16T01:08:21.494 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-16T01:08:21.874 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.868+0000 7f529b7fc700 1 Processor -- start 2023-11-16T01:08:21.874 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.868+0000 7f529b7fc700 1 -- start start 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f529b7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f529b7fc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940fc680 con 0x7f529406fc70 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5299598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5299598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53812/0 (socket says 172.21.15.133:53812) 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5299598700 1 -- 172.21.15.133:0/2472338193 learned_addr learned my addr 172.21.15.133:0/2472338193 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5299598700 1 -- 172.21.15.133:0/2472338193 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52941010f0 con 0x7f529406fc70 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5299598700 1 --2- 172.21.15.133:0/2472338193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f528400b5f0 tx=0x7f5284006ee0).ready entity=mon.0 client_cookie=fc1da2bb8adad171 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5283fff700 1 -- 172.21.15.133:0/2472338193 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f528400d600 con 0x7f529406fc70 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5283fff700 1 -- 172.21.15.133:0/2472338193 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f528400dc10 con 0x7f529406fc70 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.869+0000 7f5283fff700 1 -- 172.21.15.133:0/2472338193 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5284011cd0 con 0x7f529406fc70 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 -- 172.21.15.133:0/2472338193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 msgr2=0x7f5294100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:21.875 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2472338193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f528400b5f0 tx=0x7f5284006ee0).stop 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 -- 172.21.15.133:0/2472338193 shutdown_connections 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2472338193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f5294100c00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 -- 172.21.15.133:0/2472338193 >> 172.21.15.133:0/2472338193 conn(0x7f52940f6280 msgr2=0x7f52940f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 -- 172.21.15.133:0/2472338193 shutdown_connections 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 -- 172.21.15.133:0/2472338193 wait complete. 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.870+0000 7f529b7fc700 1 Processor -- start 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f529b7fc700 1 -- start start 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f529b7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f529b7fc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529410d1f0 con 0x7f529406fc70 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5299598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5299598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53828/0 (socket says 172.21.15.133:53828) 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5299598700 1 -- 172.21.15.133:0/2078055448 learned_addr learned my addr 172.21.15.133:0/2078055448 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:21.876 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5299598700 1 -- 172.21.15.133:0/2078055448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528400b200 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5299598700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f528400dfb0 tx=0x7f5284006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5284011df0 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f528401b8c0 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529410d3f0 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f528401a800 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.871+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529410d800 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.872+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f528400f070 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.872+0000 7f5281ffb700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 0x7f527c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.872+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5284041e30 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.872+0000 7f5298d97700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 0x7f527c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.873+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5294045ad0 con 0x7f529406fc70 2023-11-16T01:08:21.877 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.873+0000 7f5298d97700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 0x7f527c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f52900096a0 tx=0x7f5290006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:21.878 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:21.877+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5284016070 con 0x7f529406fc70 2023-11-16T01:08:22.056 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.054+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 --> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f529410ecf0 con 0x7f527c02feb0 2023-11-16T01:08:22.062 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.061+0000 7f5281ffb700 1 -- 172.21.15.133:0/2078055448 <== mgr.14142 v2:172.21.15.133:6800/3633665222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f529410ecf0 con 0x7f527c02feb0 2023-11-16T01:08:22.063 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 msgr2=0x7f527c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 0x7f527c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f52900096a0 tx=0x7f5290006b40).stop 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 msgr2=0x7f529410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f528400dfb0 tx=0x7f5284006ee0).stop 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 shutdown_connections 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f527c02feb0 0x7f527c032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 --2- 172.21.15.133:0/2078055448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f529406fc70 0x7f529410cd00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:22.066 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 >> 172.21.15.133:0/2078055448 conn(0x7f52940f6280 msgr2=0x7f52940f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:22.066 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 shutdown_connections 2023-11-16T01:08:22.066 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.063+0000 7f529b7fc700 1 -- 172.21.15.133:0/2078055448 wait complete. 2023-11-16T01:08:22.225 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:20.139957+0000 mgr.smithi133.xdyttl (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:22.225 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:20.140473+0000 mgr.smithi133.xdyttl (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-16T01:08:22.225 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:21.089426+0000 mon.smithi133 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:22.418 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-16T01:08:22.791 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0be0a23700 1 Processor -- start 2023-11-16T01:08:22.791 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0be0a23700 1 -- start start 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0be0a23700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0be0a23700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0fed60 con 0x7f0bdc06fc90 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53834/0 (socket says 172.21.15.133:53834) 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0bda59c700 1 -- 172.21.15.133:0/1360199245 learned_addr learned my addr 172.21.15.133:0/1360199245 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:22.792 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0bda59c700 1 -- 172.21.15.133:0/1360199245 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0feea0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.784+0000 7f0bda59c700 1 --2- 172.21.15.133:0/1360199245 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0bd000b5f0 tx=0x7f0bd0006ee0).ready entity=mon.0 client_cookie=b1177d0435c5f1ff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0bd959a700 1 -- 172.21.15.133:0/1360199245 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd000d600 con 0x7f0bdc06fc90 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0bd959a700 1 -- 172.21.15.133:0/1360199245 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd000dc10 con 0x7f0bdc06fc90 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0bd959a700 1 -- 172.21.15.133:0/1360199245 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd0011cd0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 -- 172.21.15.133:0/1360199245 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 msgr2=0x7f0bdc0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 --2- 172.21.15.133:0/1360199245 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0bd000b5f0 tx=0x7f0bd0006ee0).stop 2023-11-16T01:08:22.793 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 -- 172.21.15.133:0/1360199245 shutdown_connections 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 --2- 172.21.15.133:0/1360199245 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc0fe7e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 -- 172.21.15.133:0/1360199245 >> 172.21.15.133:0/1360199245 conn(0x7f0bdc0f6280 msgr2=0x7f0bdc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 -- 172.21.15.133:0/1360199245 shutdown_connections 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.785+0000 7f0be0a23700 1 -- 172.21.15.133:0/1360199245 wait complete. 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0be0a23700 1 Processor -- start 2023-11-16T01:08:22.794 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0be0a23700 1 -- start start 2023-11-16T01:08:22.795 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0be0a23700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:22.795 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0be0a23700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc10af30 con 0x7f0bdc06fc90 2023-11-16T01:08:22.795 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53836/0 (socket says 172.21.15.133:53836) 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0bda59c700 1 -- 172.21.15.133:0/49894325 learned_addr learned my addr 172.21.15.133:0/49894325 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0bda59c700 1 -- 172.21.15.133:0/49894325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd000b200 con 0x7f0bdc06fc90 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.786+0000 7f0bda59c700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0bd000dfb0 tx=0x7f0bd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd0011df0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.796 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd001b8c0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd001a7c0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc10b130 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc10b540 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0bd000f070 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.787+0000 7f0bcaffd700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 0x7f0bc0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.788+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0bd0040fc0 con 0x7f0bdc06fc90 2023-11-16T01:08:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.788+0000 7f0bd9d9b700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 0x7f0bc0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:22.798 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.788+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc059100 con 0x7f0bdc06fc90 2023-11-16T01:08:22.798 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.788+0000 7f0bd9d9b700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 0x7f0bc0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0bcc0096a0 tx=0x7f0bcc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:22.799 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.792+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bd0016080 con 0x7f0bdc06fc90 2023-11-16T01:08:22.946 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.944+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f0bdc104890 con 0x7f0bdc06fc90 2023-11-16T01:08:22.951 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.950+0000 7f0bcaffd700 1 -- 172.21.15.133:0/49894325 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0bd0019770 con 0x7f0bdc06fc90 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 msgr2=0x7f0bc0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 0x7f0bc0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0bcc0096a0 tx=0x7f0bcc006b40).stop 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 msgr2=0x7f0bdc10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0bd000dfb0 tx=0x7f0bd0006ee0).stop 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 shutdown_connections 2023-11-16T01:08:22.954 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0bc002feb0 0x7f0bc0032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:22.955 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 --2- 172.21.15.133:0/49894325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0bdc06fc90 0x7f0bdc10aa40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:22.955 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 >> 172.21.15.133:0/49894325 conn(0x7f0bdc0f6280 msgr2=0x7f0bdc0f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:22.955 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 shutdown_connections 2023-11-16T01:08:22.955 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:22.952+0000 7f0be0a23700 1 -- 172.21.15.133:0/49894325 wait complete. 2023-11-16T01:08:23.109 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:21.086637+0000 mgr.smithi133.xdyttl (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:23.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:21.087326+0000 mgr.smithi133.xdyttl (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-16T01:08:23.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:22.062377+0000 mon.smithi133 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.133:0/382182384' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:23.293 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-16T01:08:23.664 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb7111f5700 1 Processor -- start 2023-11-16T01:08:23.664 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb7111f5700 1 -- start start 2023-11-16T01:08:23.664 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb7111f5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb7111f5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0fb480 con 0x7fb70c0faae0 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb70ad9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb70ad9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54180/0 (socket says 172.21.15.133:54180) 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb70ad9d700 1 -- 172.21.15.133:0/246343331 learned_addr learned my addr 172.21.15.133:0/246343331 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.656+0000 7fb70ad9d700 1 -- 172.21.15.133:0/246343331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0fb5c0 con 0x7fb70c0faae0 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb70ad9d700 1 --2- 172.21.15.133:0/246343331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb6fc00b870 tx=0x7fb6fc006f70).ready entity=mon.0 client_cookie=15bb3e5399051694 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:23.665 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb709d9b700 1 -- 172.21.15.133:0/246343331 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6fc00da20 con 0x7fb70c0faae0 2023-11-16T01:08:23.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb709d9b700 1 -- 172.21.15.133:0/246343331 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6fc010470 con 0x7fb70c0faae0 2023-11-16T01:08:23.666 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb709d9b700 1 -- 172.21.15.133:0/246343331 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6fc00f5b0 con 0x7fb70c0faae0 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 -- 172.21.15.133:0/246343331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 msgr2=0x7fb70c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 --2- 172.21.15.133:0/246343331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb6fc00b870 tx=0x7fb6fc006f70).stop 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 -- 172.21.15.133:0/246343331 shutdown_connections 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 --2- 172.21.15.133:0/246343331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c0faf00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 -- 172.21.15.133:0/246343331 >> 172.21.15.133:0/246343331 conn(0x7fb70c0f6280 msgr2=0x7fb70c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:23.667 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 -- 172.21.15.133:0/246343331 shutdown_connections 2023-11-16T01:08:23.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.657+0000 7fb7111f5700 1 -- 172.21.15.133:0/246343331 wait complete. 2023-11-16T01:08:23.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb7111f5700 1 Processor -- start 2023-11-16T01:08:23.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb7111f5700 1 -- start start 2023-11-16T01:08:23.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb7111f5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:23.668 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb7111f5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c108d10 con 0x7fb70c0faae0 2023-11-16T01:08:23.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb70ad9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:23.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb70ad9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54196/0 (socket says 172.21.15.133:54196) 2023-11-16T01:08:23.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.658+0000 7fb70ad9d700 1 -- 172.21.15.133:0/3683317061 learned_addr learned my addr 172.21.15.133:0/3683317061 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:23.669 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb70ad9d700 1 -- 172.21.15.133:0/3683317061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc00b480 con 0x7fb70c0faae0 2023-11-16T01:08:23.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb70ad9d700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb6fc00dd70 tx=0x7fb6fc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:23.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6fc00f3d0 con 0x7fb70c0faae0 2023-11-16T01:08:23.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6fc00fea0 con 0x7fb70c0faae0 2023-11-16T01:08:23.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c108f10 con 0x7fb70c0faae0 2023-11-16T01:08:23.671 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6fc019ca0 con 0x7fb70c0faae0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.659+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c109320 con 0x7fb70c0faae0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb6fc016030 con 0x7fb70c0faae0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb6f37fe700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 0x7fb6f8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb6fc044610 con 0x7fb70c0faae0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb70a59c700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 0x7fb6f8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c059100 con 0x7fb70c0faae0 2023-11-16T01:08:23.672 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.660+0000 7fb70a59c700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 0x7fb6f8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb7000096a0 tx=0x7fb700006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:23.673 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.664+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6fc014070 con 0x7fb70c0faae0 2023-11-16T01:08:23.815 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.814+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb70c102500 con 0x7fb70c0faae0 2023-11-16T01:08:23.818 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.817+0000 7fb6f37fe700 1 -- 172.21.15.133:0/3683317061 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb6fc024420 con 0x7fb70c0faae0 2023-11-16T01:08:23.820 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.818+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 msgr2=0x7fb6f8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:23.821 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.818+0000 7fb7111f5700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 0x7fb6f8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb7000096a0 tx=0x7fb700006b40).stop 2023-11-16T01:08:23.821 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.818+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 msgr2=0x7fb70c108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:23.821 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.818+0000 7fb7111f5700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb6fc00dd70 tx=0x7fb6fc006f70).stop 2023-11-16T01:08:23.821 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 shutdown_connections 2023-11-16T01:08:23.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7fb6f802fe60 0x7fb6f8032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:23.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 --2- 172.21.15.133:0/3683317061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb70c0faae0 0x7fb70c108820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:23.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 >> 172.21.15.133:0/3683317061 conn(0x7fb70c0f6280 msgr2=0x7fb70c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:23.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 shutdown_connections 2023-11-16T01:08:23.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:23.819+0000 7fb7111f5700 1 -- 172.21.15.133:0/3683317061 wait complete. 2023-11-16T01:08:24.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:22.056744+0000 mgr.smithi133.xdyttl (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:24.138 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:22.057212+0000 mgr.smithi133.xdyttl (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-16T01:08:24.138 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:22.951512+0000 mon.smithi133 (mon.0) 85 : audit [INF] from='client.? 172.21.15.133:0/49894325' entity='client.admin' 2023-11-16T01:08:24.138 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:23.818275+0000 mon.smithi133 (mon.0) 86 : audit [INF] from='client.? 172.21.15.133:0/3683317061' entity='client.admin' 2023-11-16T01:08:24.185 INFO:teuthology.orchestra.run.smithi133.stderr:Enabling the dashboard module... 2023-11-16T01:08:24.186 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.530+0000 7f0c8b4db700 1 Processor -- start 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.530+0000 7f0c8b4db700 1 -- start start 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c8b4db700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c8b4db700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840720d0 con 0x7f0c84071cb0 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c89277700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c89277700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54200/0 (socket says 172.21.15.133:54200) 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c89277700 1 -- 172.21.15.133:0/3790798129 learned_addr learned my addr 172.21.15.133:0/3790798129 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c89277700 1 -- 172.21.15.133:0/3790798129 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84070740 con 0x7f0c84071cb0 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c89277700 1 --2- 172.21.15.133:0/3790798129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0c7800b5f0 tx=0x7f0c78006ee0).ready entity=mon.0 client_cookie=425508be300f0b8f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:24.535 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c77fff700 1 -- 172.21.15.133:0/3790798129 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7800d600 con 0x7f0c84071cb0 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c77fff700 1 -- 172.21.15.133:0/3790798129 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c7800dc10 con 0x7f0c84071cb0 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.531+0000 7f0c77fff700 1 -- 172.21.15.133:0/3790798129 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c78011cd0 con 0x7f0c84071cb0 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.532+0000 7f0c8b4db700 1 -- 172.21.15.133:0/3790798129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 msgr2=0x7f0c84070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.532+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/3790798129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0c7800b5f0 tx=0x7f0c78006ee0).stop 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.532+0000 7f0c8b4db700 1 -- 172.21.15.133:0/3790798129 shutdown_connections 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.532+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/3790798129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c84070250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.532+0000 7f0c8b4db700 1 -- 172.21.15.133:0/3790798129 >> 172.21.15.133:0/3790798129 conn(0x7f0c840f73f0 msgr2=0x7f0c840f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 -- 172.21.15.133:0/3790798129 shutdown_connections 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 -- 172.21.15.133:0/3790798129 wait complete. 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 Processor -- start 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 -- start start 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:24.536 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c8b4db700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8410d210 con 0x7f0c84071cb0 2023-11-16T01:08:24.537 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c89277700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:24.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c89277700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54208/0 (socket says 172.21.15.133:54208) 2023-11-16T01:08:24.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.533+0000 7f0c89277700 1 -- 172.21.15.133:0/751913223 learned_addr learned my addr 172.21.15.133:0/751913223 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:24.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c89277700 1 -- 172.21.15.133:0/751913223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7800b200 con 0x7f0c84071cb0 2023-11-16T01:08:24.538 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c89277700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0c78006800 tx=0x7f0c78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c78011df0 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c7801b8c0 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8410d410 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7801a830 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.534+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8410d820 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.535+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0c7800f070 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.535+0000 7f0c75ffb700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 0x7f0c6c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.535+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0c78040fe0 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.535+0000 7f0c88a76700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 0x7f0c6c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.535+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c84045ad0 con 0x7f0c84071cb0 2023-11-16T01:08:24.539 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.536+0000 7f0c88a76700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 0x7f0c6c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0c800096a0 tx=0x7f0c80006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:24.541 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.540+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c78016070 con 0x7f0c84071cb0 2023-11-16T01:08:24.725 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.724+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0c84101dc0 con 0x7f0c84071cb0 2023-11-16T01:08:24.848 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.847+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0c78024790 con 0x7f0c84071cb0 2023-11-16T01:08:24.849 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.848+0000 7f0c75ffb700 1 -- 172.21.15.133:0/751913223 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0c78019950 con 0x7f0c84071cb0 2023-11-16T01:08:24.851 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.849+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 msgr2=0x7f0c6c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:24.851 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.849+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 0x7f0c6c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0c800096a0 tx=0x7f0c80006b40).stop 2023-11-16T01:08:24.851 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.849+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 msgr2=0x7f0c8410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:24.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.849+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0c78006800 tx=0x7f0c78006ee0).stop 2023-11-16T01:08:24.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 shutdown_connections 2023-11-16T01:08:24.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f0c6c02feb0 0x7f0c6c032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:24.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 --2- 172.21.15.133:0/751913223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c84071cb0 0x7f0c8410cd20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:24.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 >> 172.21.15.133:0/751913223 conn(0x7f0c840f73f0 msgr2=0x7f0c840f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:24.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 shutdown_connections 2023-11-16T01:08:24.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:24.852+0000 7f0c8b4db700 1 -- 172.21.15.133:0/751913223 wait complete. 2023-11-16T01:08:25.178 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:24 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:24.725869+0000 mon.smithi133 (mon.0) 87 : audit [INF] from='client.? 172.21.15.133:0/751913223' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T01:08:25.252 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.674+0000 7f805590a700 1 Processor -- start 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f805590a700 1 -- start start 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f805590a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f805590a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80500fb3e0 con 0x7f80500faad0 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f804effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:25.677 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f804effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54238/0 (socket says 172.21.15.133:54238) 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f804effd700 1 -- 172.21.15.133:0/3908867108 learned_addr learned my addr 172.21.15.133:0/3908867108 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f804effd700 1 -- 172.21.15.133:0/3908867108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80500fb520 con 0x7f80500faad0 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.675+0000 7f804effd700 1 --2- 172.21.15.133:0/3908867108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f804000b870 tx=0x7f8040006f70).ready entity=mon.0 client_cookie=7016887bee5d681 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.676+0000 7f804dffb700 1 -- 172.21.15.133:0/3908867108 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804000da20 con 0x7f80500faad0 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.676+0000 7f804dffb700 1 -- 172.21.15.133:0/3908867108 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8040010470 con 0x7f80500faad0 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.676+0000 7f804dffb700 1 -- 172.21.15.133:0/3908867108 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804000f5b0 con 0x7f80500faad0 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.676+0000 7f805590a700 1 -- 172.21.15.133:0/3908867108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 msgr2=0x7f80500faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:25.679 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.676+0000 7f805590a700 1 --2- 172.21.15.133:0/3908867108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f804000b870 tx=0x7f8040006f70).stop 2023-11-16T01:08:25.680 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.677+0000 7f805590a700 1 -- 172.21.15.133:0/3908867108 shutdown_connections 2023-11-16T01:08:25.680 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.677+0000 7f805590a700 1 --2- 172.21.15.133:0/3908867108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f80500faef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:25.680 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.677+0000 7f805590a700 1 -- 172.21.15.133:0/3908867108 >> 172.21.15.133:0/3908867108 conn(0x7f80500f6310 msgr2=0x7f80500f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:25.680 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.677+0000 7f805590a700 1 -- 172.21.15.133:0/3908867108 shutdown_connections 2023-11-16T01:08:25.680 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.677+0000 7f805590a700 1 -- 172.21.15.133:0/3908867108 wait complete. 2023-11-16T01:08:25.682 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f805590a700 1 Processor -- start 2023-11-16T01:08:25.682 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f805590a700 1 -- start start 2023-11-16T01:08:25.682 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f805590a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:25.682 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f805590a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8050108c70 con 0x7f80500faad0 2023-11-16T01:08:25.682 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f804effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f804effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54250/0 (socket says 172.21.15.133:54250) 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.678+0000 7f804effd700 1 -- 172.21.15.133:0/528816128 learned_addr learned my addr 172.21.15.133:0/528816128 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f804effd700 1 -- 172.21.15.133:0/528816128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804000b480 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f804effd700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f804000b840 tx=0x7f8040006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804000f3d0 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8050108e70 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f804000fef0 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8040019c70 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.679+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8050109280 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8040016030 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8050045ad0 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f80377fe700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 0x7f803c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f80400446a0 con 0x7f80500faad0 2023-11-16T01:08:25.683 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f804e7fc700 1 -- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 msgr2=0x7f803c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:25.684 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.680+0000 7f804e7fc700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 0x7f803c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T01:08:25.685 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.684+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8040022780 con 0x7f80500faad0 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.878+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f805010a330 con 0x7f80500faad0 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.879+0000 7f80377fe700 1 -- 172.21.15.133:0/528816128 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f8040014070 con 0x7f80500faad0 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "epoch": 12, 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "available": true, 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "active_name": "smithi133.xdyttl", 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-16T01:08:25.881 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:08:25.882 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 msgr2=0x7f803c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 0x7f803c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 msgr2=0x7f8050108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f804000b840 tx=0x7f8040006f70).stop 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 shutdown_connections 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7f803c02feb0 0x7f803c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 --2- 172.21.15.133:0/528816128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80500faad0 0x7f8050108780 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 >> 172.21.15.133:0/528816128 conn(0x7f80500f6310 msgr2=0x7f80500ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 shutdown_connections 2023-11-16T01:08:25.883 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:25.880+0000 7f805590a700 1 -- 172.21.15.133:0/528816128 wait complete. 2023-11-16T01:08:26.204 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:24.849338+0000 mon.smithi133 (mon.0) 88 : audit [INF] from='client.? 172.21.15.133:0/751913223' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T01:08:26.204 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:24.849384+0000 mon.smithi133 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi133.xdyttl(active, since 7s) 2023-11-16T01:08:26.242 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for the mgr to restart... 2023-11-16T01:08:26.243 INFO:teuthology.orchestra.run.smithi133.stderr:Waiting for mgr epoch 12... 2023-11-16T01:08:26.243 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-16T01:08:26.618 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.615+0000 7ff715b53700 1 Processor -- start 2023-11-16T01:08:26.618 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.615+0000 7ff715b53700 1 -- start start 2023-11-16T01:08:26.618 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.615+0000 7ff715b53700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.615+0000 7ff715b53700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7100fc4d0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54260/0 (socket says 172.21.15.133:54260) 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70f7fe700 1 -- 172.21.15.133:0/3860022682 learned_addr learned my addr 172.21.15.133:0/3860022682 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70f7fe700 1 -- 172.21.15.133:0/3860022682 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7100fc610 con 0x7ff7100fbbc0 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70f7fe700 1 --2- 172.21.15.133:0/3860022682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff700009960 tx=0x7ff7000092d0).ready entity=mon.0 client_cookie=394b6d7a28d661e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:26.619 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70e7fc700 1 -- 172.21.15.133:0/3860022682 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70000b9c0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.620 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70e7fc700 1 -- 172.21.15.133:0/3860022682 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff700011400 con 0x7ff7100fbbc0 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.616+0000 7ff70e7fc700 1 -- 172.21.15.133:0/3860022682 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff700010580 con 0x7ff7100fbbc0 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 -- 172.21.15.133:0/3860022682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 msgr2=0x7ff7100fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 --2- 172.21.15.133:0/3860022682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff700009960 tx=0x7ff7000092d0).stop 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 -- 172.21.15.133:0/3860022682 shutdown_connections 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 --2- 172.21.15.133:0/3860022682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff7100fbfe0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 -- 172.21.15.133:0/3860022682 >> 172.21.15.133:0/3860022682 conn(0x7ff7100f73e0 msgr2=0x7ff7100f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 -- 172.21.15.133:0/3860022682 shutdown_connections 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.617+0000 7ff715b53700 1 -- 172.21.15.133:0/3860022682 wait complete. 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff715b53700 1 Processor -- start 2023-11-16T01:08:26.621 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff715b53700 1 -- start start 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff715b53700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff715b53700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71010af20 con 0x7ff7100fbbc0 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff70f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff70f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54264/0 (socket says 172.21.15.133:54264) 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.618+0000 7ff70f7fe700 1 -- 172.21.15.133:0/1228183910 learned_addr learned my addr 172.21.15.133:0/1228183910 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff70f7fe700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff700009570 con 0x7ff7100fbbc0 2023-11-16T01:08:26.622 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff70f7fe700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff700006dc0 tx=0x7ff7000092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:26.623 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7000103a0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.623 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71010b120 con 0x7ff7100fbbc0 2023-11-16T01:08:26.623 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff700010cf0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.623 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70001baf0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.623 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.619+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71010b530 con 0x7ff7100fbbc0 2023-11-16T01:08:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff70000f070 con 0x7ff7100fbbc0 2023-11-16T01:08:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff707fff700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6f0032a30 con 0x7ff6f002feb0 2023-11-16T01:08:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff700041af0 con 0x7ff7100fbbc0 2023-11-16T01:08:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff70effd700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.620+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-16T01:08:26.822 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.821+0000 7ff70effd700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:26.823 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:26.821+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-16T01:08:27.224 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:27.222+0000 7ff70effd700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:27.225 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:27.222+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-16T01:08:27.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:26 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:25.880221+0000 mon.smithi133 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.133:0/528816128' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T01:08:28.024 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:28.023+0000 7ff70effd700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:28.025 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:28.023+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-16T01:08:29.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:29.624+0000 7ff70effd700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3633665222 2023-11-16T01:08:29.626 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:29.624+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-16T01:08:29.928 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:29.926+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7ff700022d80 con 0x7ff7100fbbc0 2023-11-16T01:08:29.928 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:29.926+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 msgr2=0x7ff6f0032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T01:08:29.928 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:29.926+0000 7ff707fff700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:30.930 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.927+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7ff700010a20 con 0x7ff7100fbbc0 2023-11-16T01:08:30.930 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.928+0000 7ff707fff700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 0x7ff6f0035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:30.930 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.928+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6f0032a30 con 0x7ff6f0033b40 2023-11-16T01:08:30.930 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.928+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 0x7ff6f0035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:30.930 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.928+0000 7ff70effd700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 0x7ff6f0035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff6f8006fd0 tx=0x7ff6f8006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:30.931 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.929+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff6f0032a30 con 0x7ff6f0033b40 2023-11-16T01:08:30.940 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.938+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff710059100 con 0x7ff6f0033b40 2023-11-16T01:08:30.940 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.938+0000 7ff707fff700 1 -- 172.21.15.133:0/1228183910 <== mgr.14162 v2:172.21.15.133:6800/2350789018 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff710059100 con 0x7ff6f0033b40 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: { 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: "initialized": true 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: } 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 msgr2=0x7ff6f0035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 0x7ff6f0035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff6f8006fd0 tx=0x7ff6f8006cf0).stop 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 msgr2=0x7ff71010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff700006dc0 tx=0x7ff7000092d0).stop 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 shutdown_connections 2023-11-16T01:08:30.942 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff6f0033b40 0x7ff6f0035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:30.943 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7100fbbc0 0x7ff71010aa30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:30.943 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 --2- 172.21.15.133:0/1228183910 >> [v2:172.21.15.133:6800/3633665222,v1:172.21.15.133:6801/3633665222] conn(0x7ff6f002feb0 0x7ff6f0032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-16T01:08:30.943 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 >> 172.21.15.133:0/1228183910 conn(0x7ff7100f73e0 msgr2=0x7ff7100f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:30.943 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 shutdown_connections 2023-11-16T01:08:30.943 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:30.939+0000 7ff715b53700 1 -- 172.21.15.133:0/1228183910 wait complete. 2023-11-16T01:08:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:29.875336+0000 mon.smithi133 (mon.0) 91 : cluster [INF] Active manager daemon smithi133.xdyttl restarted 2023-11-16T01:08:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:29.875801+0000 mon.smithi133 (mon.0) 92 : cluster [INF] Activating manager daemon smithi133.xdyttl 2023-11-16T01:08:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:29.879801+0000 mon.smithi133 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-16T01:08:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:29.928348+0000 mon.smithi133 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi133.xdyttl(active, starting, since 0.0526746s) 2023-11-16T01:08:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:29.929294+0000 mon.smithi133 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2023-11-16T01:08:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:29.929878+0000 mon.smithi133 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mgr metadata", "who": "smithi133.xdyttl", "id": "smithi133.xdyttl"}]: dispatch 2023-11-16T01:08:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:29.930683+0000 mon.smithi133 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:29.930840+0000 mon.smithi133 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:29.930974+0000 mon.smithi133 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:29.944871+0000 mon.smithi133 (mon.0) 100 : cluster [INF] Manager daemon smithi133.xdyttl is now available 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.042706+0000 mon.smithi133 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.043769+0000 mon.smithi133 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.068380+0000 mon.smithi133 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:08:31.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.071118+0000 mon.smithi133 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:08:31.284 INFO:teuthology.orchestra.run.smithi133.stderr:mgr epoch 12 is available 2023-11-16T01:08:31.284 INFO:teuthology.orchestra.run.smithi133.stderr:Generating a dashboard self-signed certificate... 2023-11-16T01:08:31.285 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-16T01:08:31.743 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.728+0000 7f397ba5c700 1 Processor -- start 2023-11-16T01:08:31.748 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.728+0000 7f397ba5c700 1 -- start start 2023-11-16T01:08:31.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.728+0000 7f397ba5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:31.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.728+0000 7f397ba5c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740fdb80 con 0x7f39740fd270 2023-11-16T01:08:31.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f39797f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:31.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f39797f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54324/0 (socket says 172.21.15.133:54324) 2023-11-16T01:08:31.852 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f39797f8700 1 -- 172.21.15.133:0/2783516673 learned_addr learned my addr 172.21.15.133:0/2783516673 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f39797f8700 1 -- 172.21.15.133:0/2783516673 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740fdcc0 con 0x7f39740fd270 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f39797f8700 1 --2- 172.21.15.133:0/2783516673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f397000b5f0 tx=0x7f3970006ee0).ready entity=mon.0 client_cookie=e0f8a29ac317c90f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f3963fff700 1 -- 172.21.15.133:0/2783516673 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f397000d600 con 0x7f39740fd270 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f3963fff700 1 -- 172.21.15.133:0/2783516673 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f397000dc10 con 0x7f39740fd270 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.729+0000 7f3963fff700 1 -- 172.21.15.133:0/2783516673 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3970011cd0 con 0x7f39740fd270 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 -- 172.21.15.133:0/2783516673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 msgr2=0x7f39740fd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2783516673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f397000b5f0 tx=0x7f3970006ee0).stop 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 -- 172.21.15.133:0/2783516673 shutdown_connections 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2783516673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f39740fd690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 -- 172.21.15.133:0/2783516673 >> 172.21.15.133:0/2783516673 conn(0x7f39740f73e0 msgr2=0x7f39740f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 -- 172.21.15.133:0/2783516673 shutdown_connections 2023-11-16T01:08:31.853 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.730+0000 7f397ba5c700 1 -- 172.21.15.133:0/2783516673 wait complete. 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f397ba5c700 1 Processor -- start 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f397ba5c700 1 -- start start 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f397ba5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f397ba5c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397406f0e0 con 0x7f39740fd270 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f39797f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f39797f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54330/0 (socket says 172.21.15.133:54330) 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f39797f8700 1 -- 172.21.15.133:0/2680434742 learned_addr learned my addr 172.21.15.133:0/2680434742 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f39797f8700 1 -- 172.21.15.133:0/2680434742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397000b200 con 0x7f39740fd270 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.731+0000 7f39797f8700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3970006cb0 tx=0x7f3970006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.732+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3970011df0 con 0x7f39740fd270 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.732+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f397001b8c0 con 0x7f39740fd270 2023-11-16T01:08:31.854 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.732+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397406f280 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.732+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f397001a930 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.732+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397406f690 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f397000f070 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f3961ffb700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 0x7f395c032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3974059100 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3970041010 con 0x7f39740fd270 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f3978ff7700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 0x7f395c032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.733+0000 7f3978ff7700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 0x7f395c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f39640096a0 tx=0x7f3964006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:31.855 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.737+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3970010bc0 con 0x7f39740fd270 2023-11-16T01:08:31.899 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.897+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f39740fc480 con 0x7f395c02fdc0 2023-11-16T01:08:31.968 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.966+0000 7f3961ffb700 1 -- 172.21.15.133:0/2680434742 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f39740fc480 con 0x7f395c02fdc0 2023-11-16T01:08:31.969 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 msgr2=0x7f395c032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 0x7f395c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f39640096a0 tx=0x7f3964006b40).stop 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 msgr2=0x7f3974070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3970006cb0 tx=0x7f3970006ee0).stop 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 shutdown_connections 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f395c02fdc0 0x7f395c032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 --2- 172.21.15.133:0/2680434742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39740fd270 0x7f3974070c60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.968+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 >> 172.21.15.133:0/2680434742 conn(0x7f39740f73e0 msgr2=0x7f39740f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:31.972 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.969+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 shutdown_connections 2023-11-16T01:08:31.973 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:31.969+0000 7f397ba5c700 1 -- 172.21.15.133:0/2680434742 wait complete. 2023-11-16T01:08:32.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.930386+0000 mgr.smithi133.xdyttl (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T01:08:32.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:30.930825+0000 mon.smithi133 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi133.xdyttl(active, since 1.05515s) 2023-11-16T01:08:32.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.940001+0000 mgr.smithi133.xdyttl (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T01:08:32.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:30.971570+0000 mon.smithi133 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:32.325 INFO:teuthology.orchestra.run.smithi133.stderr:Creating initial admin user... 2023-11-16T01:08:32.326 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprqfh7zkh:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-16T01:08:32.688 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.682+0000 7f16bc7fc700 1 Processor -- start 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.682+0000 7f16bc7fc700 1 -- start start 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.682+0000 7f16bc7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.682+0000 7f16bc7fc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40fed20 con 0x7f16b406fcd0 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16ba598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16ba598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54336/0 (socket says 172.21.15.133:54336) 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16ba598700 1 -- 172.21.15.133:0/2216507983 learned_addr learned my addr 172.21.15.133:0/2216507983 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16ba598700 1 -- 172.21.15.133:0/2216507983 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40fee60 con 0x7f16b406fcd0 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16ba598700 1 --2- 172.21.15.133:0/2216507983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f16a400b5f0 tx=0x7f16a4006ee0).ready entity=mon.0 client_cookie=ad1add42c8ffbb79 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:32.689 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16b9596700 1 -- 172.21.15.133:0/2216507983 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a400d600 con 0x7f16b406fcd0 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16b9596700 1 -- 172.21.15.133:0/2216507983 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16a400dc10 con 0x7f16b406fcd0 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.683+0000 7f16b9596700 1 -- 172.21.15.133:0/2216507983 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a4011cd0 con 0x7f16b406fcd0 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 -- 172.21.15.133:0/2216507983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 msgr2=0x7f16b40fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/2216507983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f16a400b5f0 tx=0x7f16a4006ee0).stop 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 -- 172.21.15.133:0/2216507983 shutdown_connections 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/2216507983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b40fe7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 -- 172.21.15.133:0/2216507983 >> 172.21.15.133:0/2216507983 conn(0x7f16b40f6280 msgr2=0x7f16b40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 -- 172.21.15.133:0/2216507983 shutdown_connections 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.684+0000 7f16bc7fc700 1 -- 172.21.15.133:0/2216507983 wait complete. 2023-11-16T01:08:32.690 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16bc7fc700 1 Processor -- start 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16bc7fc700 1 -- start start 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16bc7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16bc7fc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b410af50 con 0x7f16b406fcd0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16ba598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16ba598700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54350/0 (socket says 172.21.15.133:54350) 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16ba598700 1 -- 172.21.15.133:0/3453853608 learned_addr learned my addr 172.21.15.133:0/3453853608 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16ba598700 1 -- 172.21.15.133:0/3453853608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a400b200 con 0x7f16b406fcd0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.685+0000 7f16ba598700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f16a400dfb0 tx=0x7f16a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a4011df0 con 0x7f16b406fcd0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16a401b8c0 con 0x7f16b406fcd0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b410b150 con 0x7f16b406fcd0 2023-11-16T01:08:32.691 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a401a800 con 0x7f16b406fcd0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b410b560 con 0x7f16b406fcd0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.686+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f16a400f070 con 0x7f16b406fcd0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.687+0000 7f16a2ffd700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 0x7f169c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.687+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16a4041090 con 0x7f16b406fcd0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.687+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b4045ad0 con 0x7f16b406fcd0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.687+0000 7f16b9d97700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 0x7f169c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.687+0000 7f16b9d97700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 0x7f169c032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f16b00096a0 tx=0x7f16b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:32.692 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.691+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16a4016070 con 0x7f16b406fcd0 2023-11-16T01:08:32.861 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:32.859+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f16b40ffc00 con 0x7f169c02feb0 2023-11-16T01:08:33.078 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.076+0000 7f16a2ffd700 1 -- 172.21.15.133:0/3453853608 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f16b40ffc00 con 0x7f169c02feb0 2023-11-16T01:08:33.078 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$eXblx9ufzCswzqZEFzRJF.ugsx6k2lInn83/mA1qtbyDRpoUI8Ydm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700096913, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T01:08:33.080 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 msgr2=0x7f169c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:33.080 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 0x7f169c032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f16b00096a0 tx=0x7f16b0006b40).stop 2023-11-16T01:08:33.080 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 msgr2=0x7f16b410aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f16a400dfb0 tx=0x7f16a4006ee0).stop 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 shutdown_connections 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f169c02feb0 0x7f169c032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 --2- 172.21.15.133:0/3453853608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16b406fcd0 0x7f16b410aa60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 >> 172.21.15.133:0/3453853608 conn(0x7f16b40f6280 msgr2=0x7f16b40f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:33.081 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 shutdown_connections 2023-11-16T01:08:33.082 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.078+0000 7f16bc7fc700 1 -- 172.21.15.133:0/3453853608 wait complete. 2023-11-16T01:08:33.274 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:31.899155+0000 mgr.smithi133.xdyttl (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:33.274 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:31.965754+0000 mon.smithi133 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:33.274 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:31.967518+0000 mon.smithi133 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:33.274 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:32.618191+0000 mon.smithi133 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi133.xdyttl(active, since 2s) 2023-11-16T01:08:33.275 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:32.861482+0000 mgr.smithi133.xdyttl (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:08:33.466 INFO:teuthology.orchestra.run.smithi133.stderr:Fetching dashboard port number... 2023-11-16T01:08:33.466 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmplu0e6ap3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2fs2ar_1:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.851+0000 7facb85d7700 1 Processor -- start 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.851+0000 7facb85d7700 1 -- start start 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb85d7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb85d7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb00fc680 con 0x7facb006fc70 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb6373700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb6373700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33104/0 (socket says 172.21.15.133:33104) 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb6373700 1 -- 172.21.15.133:0/1479791145 learned_addr learned my addr 172.21.15.133:0/1479791145 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:33.856 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb6373700 1 -- 172.21.15.133:0/1479791145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb01010f0 con 0x7facb006fc70 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.852+0000 7facb6373700 1 --2- 172.21.15.133:0/1479791145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faca000b5f0 tx=0x7faca0006ee0).ready entity=mon.0 client_cookie=ea6a0ed88844d6ab server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb5371700 1 -- 172.21.15.133:0/1479791145 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca000d600 con 0x7facb006fc70 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb5371700 1 -- 172.21.15.133:0/1479791145 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faca000dc10 con 0x7facb006fc70 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb5371700 1 -- 172.21.15.133:0/1479791145 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca0011cd0 con 0x7facb006fc70 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 -- 172.21.15.133:0/1479791145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 msgr2=0x7facb0100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 --2- 172.21.15.133:0/1479791145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faca000b5f0 tx=0x7faca0006ee0).stop 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 -- 172.21.15.133:0/1479791145 shutdown_connections 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 --2- 172.21.15.133:0/1479791145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb0100c00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 -- 172.21.15.133:0/1479791145 >> 172.21.15.133:0/1479791145 conn(0x7facb00f6280 msgr2=0x7facb00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 -- 172.21.15.133:0/1479791145 shutdown_connections 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.853+0000 7facb85d7700 1 -- 172.21.15.133:0/1479791145 wait complete. 2023-11-16T01:08:33.857 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb85d7700 1 Processor -- start 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb85d7700 1 -- start start 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb85d7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb85d7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb010d1f0 con 0x7facb006fc70 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb6373700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb6373700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33116/0 (socket says 172.21.15.133:33116) 2023-11-16T01:08:33.858 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb6373700 1 -- 172.21.15.133:0/3408966208 learned_addr learned my addr 172.21.15.133:0/3408966208 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:33.859 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.854+0000 7facb6373700 1 -- 172.21.15.133:0/3408966208 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca000b200 con 0x7facb006fc70 2023-11-16T01:08:33.859 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7facb6373700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7faca000dfb0 tx=0x7faca0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca0011df0 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faca001b8c0 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb010d3f0 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca001a800 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.855+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb010d800 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.856+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb0045ad0 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.856+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7faca000f070 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.856+0000 7fac9effd700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 0x7fac98032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.856+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faca00410a0 con 0x7facb006fc70 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.856+0000 7facb5b72700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 0x7fac98032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:33.860 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.857+0000 7facb5b72700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 0x7fac98032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7facac0096a0 tx=0x7facac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:33.862 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:33.860+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faca0019950 con 0x7facb006fc70 2023-11-16T01:08:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.011+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7facb0101e90 con 0x7facb006fc70 2023-11-16T01:08:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.011+0000 7fac9effd700 1 -- 172.21.15.133:0/3408966208 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7faca0016070 con 0x7facb006fc70 2023-11-16T01:08:34.013 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 8443 2023-11-16T01:08:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 msgr2=0x7fac98032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:34.015 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 0x7fac98032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7facac0096a0 tx=0x7facac006b40).stop 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 msgr2=0x7facb010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7faca000dfb0 tx=0x7faca0006ee0).stop 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 shutdown_connections 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fac9802feb0 0x7fac98032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 --2- 172.21.15.133:0/3408966208 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7facb006fc70 0x7facb010cd00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.013+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 >> 172.21.15.133:0/3408966208 conn(0x7facb00f6280 msgr2=0x7facb00f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.014+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 shutdown_connections 2023-11-16T01:08:34.016 INFO:teuthology.orchestra.run.smithi133.stderr:/usr/bin/ceph: 2023-11-16T01:08:34.014+0000 7facb85d7700 1 -- 172.21.15.133:0/3408966208 wait complete. 2023-11-16T01:08:34.402 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:34 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:33.077697+0000 mon.smithi133 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:34.403 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:34 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:34.013079+0000 mon.smithi133 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.133:0/3408966208' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T01:08:34.408 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-16T01:08:34.418 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: disabled 2023-11-16T01:08:34.418 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: systemctl is-active firewalld.service 2023-11-16T01:08:34.423 INFO:teuthology.orchestra.run.smithi133.stderr:systemctl: inactive 2023-11-16T01:08:34.424 INFO:teuthology.orchestra.run.smithi133.stderr:firewalld.service is not enabled 2023-11-16T01:08:34.424 INFO:teuthology.orchestra.run.smithi133.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T01:08:34.426 INFO:teuthology.orchestra.run.smithi133.stderr:Ceph Dashboard is now available at: 2023-11-16T01:08:34.426 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.426 INFO:teuthology.orchestra.run.smithi133.stderr: URL: https://smithi133.front.sepia.ceph.com:8443/ 2023-11-16T01:08:34.427 INFO:teuthology.orchestra.run.smithi133.stderr: User: admin 2023-11-16T01:08:34.427 INFO:teuthology.orchestra.run.smithi133.stderr: Password: vvtrva2vlb 2023-11-16T01:08:34.427 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.427 INFO:teuthology.orchestra.run.smithi133.stderr:You can access the Ceph CLI with: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: ceph telemetry on 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr:For more information see: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr:Bootstrap complete. 2023-11-16T01:08:34.428 INFO:teuthology.orchestra.run.smithi133.stderr:Releasing lock 140568729164264 on /run/cephadm/7905ec40-841c-11ee-8dbe-212e2dc638e7.lock 2023-11-16T01:08:34.429 INFO:teuthology.orchestra.run.smithi133.stderr:Lock 140568729164264 released on /run/cephadm/7905ec40-841c-11ee-8dbe-212e2dc638e7.lock 2023-11-16T01:08:34.460 INFO:tasks.cephadm:Fetching config... 2023-11-16T01:08:34.461 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:08:34.461 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T01:08:34.480 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T01:08:34.480 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:08:34.480 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T01:08:34.538 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T01:08:34.538 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:08:34.538 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/mon.smithi133/keyring of=/dev/stdout 2023-11-16T01:08:34.610 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T01:08:34.610 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-11-16T01:08:34.610 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T01:08:34.669 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T01:08:34.669 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5aeFol3H9vaj+lHXXAdguOGz0IKq3pfRGGnwM6ADkPU1p/KR8BlsbhayYu42+dm5b6bZN2HlGLkpAuggkq69bCGpr24N7RS+jgwR/J0Tz1cX4DdMA57wraqWRQWL6OwKUHF2JlV4yUkA00z1i1XrUgW6JPJDvLOVbRowtwx58XbFjF9A96jrQ6YfgyV79LMQYno+B5sJBH4BPervxsT6h+9rPLcMr6nkwNRZC9qyTlZ9MkZ2Pi4rsOluP2vhlfVf4h84c8ygdmFdN3AjkG4UV3ruWL3DdohcaprUsbftfY6dLVyVeZtGNqyVTKut7hTsVdACpWITIxjRbR2SbsOXx03MQHxx1mBQYj+tamPcC48lITZ+aVh69Yn6UdSEoHWx6vHbc0QDMJJdt/1UeZwNO6n7h4X2OLT5ubKWfxZZ3ljeoz3k+gwTwYk4r88TRo2OahVt2KIdGks9itkzLGhqLVU1dI+GVkqni47AdOaS16bnmsnM7/+PdaAgUFw9WiM= ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T01:08:34.746 INFO:teuthology.orchestra.run.smithi133.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5aeFol3H9vaj+lHXXAdguOGz0IKq3pfRGGnwM6ADkPU1p/KR8BlsbhayYu42+dm5b6bZN2HlGLkpAuggkq69bCGpr24N7RS+jgwR/J0Tz1cX4DdMA57wraqWRQWL6OwKUHF2JlV4yUkA00z1i1XrUgW6JPJDvLOVbRowtwx58XbFjF9A96jrQ6YfgyV79LMQYno+B5sJBH4BPervxsT6h+9rPLcMr6nkwNRZC9qyTlZ9MkZ2Pi4rsOluP2vhlfVf4h84c8ygdmFdN3AjkG4UV3ruWL3DdohcaprUsbftfY6dLVyVeZtGNqyVTKut7hTsVdACpWITIxjRbR2SbsOXx03MQHxx1mBQYj+tamPcC48lITZ+aVh69Yn6UdSEoHWx6vHbc0QDMJJdt/1UeZwNO6n7h4X2OLT5ubKWfxZZ3ljeoz3k+gwTwYk4r88TRo2OahVt2KIdGks9itkzLGhqLVU1dI+GVkqni47AdOaS16bnmsnM7/+PdaAgUFw9WiM= ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:08:34.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5aeFol3H9vaj+lHXXAdguOGz0IKq3pfRGGnwM6ADkPU1p/KR8BlsbhayYu42+dm5b6bZN2HlGLkpAuggkq69bCGpr24N7RS+jgwR/J0Tz1cX4DdMA57wraqWRQWL6OwKUHF2JlV4yUkA00z1i1XrUgW6JPJDvLOVbRowtwx58XbFjF9A96jrQ6YfgyV79LMQYno+B5sJBH4BPervxsT6h+9rPLcMr6nkwNRZC9qyTlZ9MkZ2Pi4rsOluP2vhlfVf4h84c8ygdmFdN3AjkG4UV3ruWL3DdohcaprUsbftfY6dLVyVeZtGNqyVTKut7hTsVdACpWITIxjRbR2SbsOXx03MQHxx1mBQYj+tamPcC48lITZ+aVh69Yn6UdSEoHWx6vHbc0QDMJJdt/1UeZwNO6n7h4X2OLT5ubKWfxZZ3ljeoz3k+gwTwYk4r88TRo2OahVt2KIdGks9itkzLGhqLVU1dI+GVkqni47AdOaS16bnmsnM7/+PdaAgUFw9WiM= ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T01:08:34.801 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5aeFol3H9vaj+lHXXAdguOGz0IKq3pfRGGnwM6ADkPU1p/KR8BlsbhayYu42+dm5b6bZN2HlGLkpAuggkq69bCGpr24N7RS+jgwR/J0Tz1cX4DdMA57wraqWRQWL6OwKUHF2JlV4yUkA00z1i1XrUgW6JPJDvLOVbRowtwx58XbFjF9A96jrQ6YfgyV79LMQYno+B5sJBH4BPervxsT6h+9rPLcMr6nkwNRZC9qyTlZ9MkZ2Pi4rsOluP2vhlfVf4h84c8ygdmFdN3AjkG4UV3ruWL3DdohcaprUsbftfY6dLVyVeZtGNqyVTKut7hTsVdACpWITIxjRbR2SbsOXx03MQHxx1mBQYj+tamPcC48lITZ+aVh69Yn6UdSEoHWx6vHbc0QDMJJdt/1UeZwNO6n7h4X2OLT5ubKWfxZZ3ljeoz3k+gwTwYk4r88TRo2OahVt2KIdGks9itkzLGhqLVU1dI+GVkqni47AdOaS16bnmsnM7/+PdaAgUFw9WiM= ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7 2023-11-16T01:08:34.813 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 -- 172.21.15.133:0/2460100278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 msgr2=0x7faaf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 --2- 172.21.15.133:0/2460100278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf40f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7faae4009fb0 tx=0x7faae4009b30).stop 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 -- 172.21.15.133:0/2460100278 shutdown_connections 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 --2- 172.21.15.133:0/2460100278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf40f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 -- 172.21.15.133:0/2460100278 >> 172.21.15.133:0/2460100278 conn(0x7faaf40f1c70 msgr2=0x7faaf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:36.764 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 -- 172.21.15.133:0/2460100278 shutdown_connections 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.726+0000 7faafa746700 1 -- 172.21.15.133:0/2460100278 wait complete. 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.727+0000 7faafa746700 1 Processor -- start 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.727+0000 7faafa746700 1 -- start start 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.727+0000 7faafa746700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.727+0000 7faafa746700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf4106c20 con 0x7faaf40f6510 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33136/0 (socket says 172.21.15.133:33136) 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf3fff700 1 -- 172.21.15.133:0/2442974904 learned_addr learned my addr 172.21.15.133:0/2442974904 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf3fff700 1 -- 172.21.15.133:0/2442974904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae4005040 con 0x7faaf40f6510 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf3fff700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7faae4013510 tx=0x7faae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae4009320 con 0x7faaf40f6510 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faae4013f00 con 0x7faaf40f6510 2023-11-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf4106e20 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae4014d90 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.728+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf41072b0 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faae40263a0 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faaf0ff9700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 0x7faadc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf4045ad0 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faae40410c0 con 0x7faaf40f6510 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faaf37fe700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 0x7faadc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.729+0000 7faaf37fe700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 0x7faadc032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7faae80096a0 tx=0x7faae8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:36.766 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.737+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faae4012920 con 0x7faaf40f6510 2023-11-16T01:08:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.893+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7faaf40fbf60 con 0x7faaf40f6510 2023-11-16T01:08:36.897 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.896+0000 7faaf0ff9700 1 -- 172.21.15.133:0/2442974904 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7faae4012330 con 0x7faaf40f6510 2023-11-16T01:08:36.899 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 msgr2=0x7faadc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:36.899 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 0x7faadc032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7faae80096a0 tx=0x7faae8006b40).stop 2023-11-16T01:08:36.899 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 msgr2=0x7faaf4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7faae4013510 tx=0x7faae400b040).stop 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 shutdown_connections 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7faadc02ff60 0x7faadc032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 --2- 172.21.15.133:0/2442974904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faaf40f6510 0x7faaf4106730 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 >> 172.21.15.133:0/2442974904 conn(0x7faaf40f1c70 msgr2=0x7faaf40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 shutdown_connections 2023-11-16T01:08:36.900 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:08:36.898+0000 7faafa746700 1 -- 172.21.15.133:0/2442974904 wait complete. 2023-11-16T01:08:37.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:35.941898+0000 mon.smithi133 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi133.xdyttl(active, since 6s) 2023-11-16T01:08:37.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:36.897249+0000 mon.smithi133 (mon.0) 113 : audit [INF] from='client.? 172.21.15.133:0/2442974904' entity='client.admin' 2023-11-16T01:08:37.339 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2023-11-16T01:08:37.339 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:08:37.339 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2023-11-16T01:08:37.361 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2023-11-16T01:08:37.361 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T01:08:37.422 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2023-11-16T01:08:37.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi134 2023-11-16T01:08:39.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:38.009778+0000 mon.smithi133 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:39.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:38.313914+0000 mon.smithi133 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:08:39.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:38.320244+0000 mgr.smithi133.xdyttl (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi133 on smithi133 2023-11-16T01:08:51.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:49.931940+0000 mgr.smithi133.xdyttl (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:08:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:51.932203+0000 mgr.smithi133.xdyttl (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:08:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:53.932527+0000 mgr.smithi133.xdyttl (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:08:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:55.932838+0000 mgr.smithi133.xdyttl (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:08:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1455176445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 msgr2=0x7fbb480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1455176445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb480f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbb44009fb0 tx=0x7fbb44009b30).stop 2023-11-16T01:08:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1455176445 shutdown_connections 2023-11-16T01:08:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1455176445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb480f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1455176445 >> 172.21.15.134:0/1455176445 conn(0x7fbb480f0b70 msgr2=0x7fbb480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1455176445 shutdown_connections 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1455176445 wait complete. 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 Processor -- start 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.143+0000 7fbb4f6e4700 1 -- start start 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4f6e4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4f6e4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48106c00 con 0x7fbb480f5410 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4d480700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4d480700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35706/0 (socket says 172.21.15.134:35706) 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4d480700 1 -- 172.21.15.134:0/1759310358 learned_addr learned my addr 172.21.15.134:0/1759310358 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4d480700 1 -- 172.21.15.134:0/1759310358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44005040 con 0x7fbb480f5410 2023-11-16T01:08:58.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4d480700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbb44013510 tx=0x7fbb4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:58.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb44009320 con 0x7fbb480f5410 2023-11-16T01:08:58.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb44013f00 con 0x7fbb480f5410 2023-11-16T01:08:58.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb48106e00 con 0x7fbb480f5410 2023-11-16T01:08:58.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.144+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb44014d90 con 0x7fbb480f5410 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.145+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb48107290 con 0x7fbb480f5410 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.145+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb44014420 con 0x7fbb480f5410 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.145+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48045ad0 con 0x7fbb480f5410 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.146+0000 7fbb3dffb700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.146+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb440410d0 con 0x7fbb480f5410 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.146+0000 7fbb4cc7f700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:08:58.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.146+0000 7fbb4cc7f700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 0x7fbb34032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbb380096a0 tx=0x7fbb38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:08:58.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.150+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb440095f0 con 0x7fbb480f5410 2023-11-16T01:08:58.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.321+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb480f29b0 con 0x7fbb3402ff60 2023-11-16T01:08:58.937 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi134' 2023-11-16T01:08:58.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.936+0000 7fbb3dffb700 1 -- 172.21.15.134:0/1759310358 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fbb480f29b0 con 0x7fbb3402ff60 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 msgr2=0x7fbb34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 0x7fbb34032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbb380096a0 tx=0x7fbb38006b40).stop 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 msgr2=0x7fbb48106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbb44013510 tx=0x7fbb4400b040).stop 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 shutdown_connections 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 --2- 172.21.15.134:0/1759310358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb480f5410 0x7fbb48106710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 >> 172.21.15.134:0/1759310358 conn(0x7fbb480f0b70 msgr2=0x7fbb4805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 shutdown_connections 2023-11-16T01:08:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:08:58.938+0000 7fbb4f6e4700 1 -- 172.21.15.134:0/1759310358 wait complete. 2023-11-16T01:08:59.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T01:09:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:57.933199+0000 mgr.smithi133.xdyttl (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:58.322681+0000 mgr.smithi133.xdyttl (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:09:00.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:08:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:08:58.935573+0000 mon.smithi133 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:09:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- 172.21.15.134:0/2649003831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 msgr2=0x7f8af80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 --2- 172.21.15.134:0/2649003831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af80f66c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8ae8009fb0 tx=0x7f8ae8009b30).stop 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- 172.21.15.134:0/2649003831 shutdown_connections 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 --2- 172.21.15.134:0/2649003831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af80f66c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- 172.21.15.134:0/2649003831 >> 172.21.15.134:0/2649003831 conn(0x7f8af80f1aa0 msgr2=0x7f8af80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- 172.21.15.134:0/2649003831 shutdown_connections 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- 172.21.15.134:0/2649003831 wait complete. 2023-11-16T01:09:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 Processor -- start 2023-11-16T01:09:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.425+0000 7f8afd577700 1 -- start start 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8afd577700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8afd577700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af81069a0 con 0x7f8af80f62a0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8af7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8af7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36590/0 (socket says 172.21.15.134:36590) 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8af7fff700 1 -- 172.21.15.134:0/1947711422 learned_addr learned my addr 172.21.15.134:0/1947711422 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8af7fff700 1 -- 172.21.15.134:0/1947711422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae8005040 con 0x7f8af80f62a0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.426+0000 7f8af7fff700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8ae8013510 tx=0x7f8ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae8009320 con 0x7f8af80f62a0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ae8013f00 con 0x7f8af80f62a0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af8106ba0 con 0x7f8af80f62a0 2023-11-16T01:09:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae8014dc0 con 0x7f8af80f62a0 2023-11-16T01:09:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af8107030 con 0x7f8af80f62a0 2023-11-16T01:09:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.427+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ae8014420 con 0x7f8af80f62a0 2023-11-16T01:09:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.428+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af8045ad0 con 0x7f8af80f62a0 2023-11-16T01:09:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.428+0000 7f8af4ff9700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 0x7f8ae0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.428+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ae80410a0 con 0x7f8af80f62a0 2023-11-16T01:09:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.428+0000 7f8af77fe700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 0x7f8ae0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.428+0000 7f8af77fe700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 0x7f8ae0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8aec0096a0 tx=0x7f8aec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.432+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ae806d2b0 con 0x7f8af80f62a0 2023-11-16T01:09:00.605 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.604+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8af80f2c20 con 0x7f8ae002ffb0 2023-11-16T01:09:00.606 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.606+0000 7f8af4ff9700 1 -- 172.21.15.134:0/1947711422 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f8af80f2c20 con 0x7f8ae002ffb0 2023-11-16T01:09:00.607 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:00.607 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.133", "hostname": "smithi133", "labels": [], "status": ""}, {"addr": "smithi134", "hostname": "smithi134", "labels": [], "status": ""}] 2023-11-16T01:09:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 msgr2=0x7f8ae0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 0x7f8ae0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8aec0096a0 tx=0x7f8aec006b40).stop 2023-11-16T01:09:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 msgr2=0x7f8af81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:00.608 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8ae8013510 tx=0x7f8ae800b040).stop 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 shutdown_connections 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ae002ffb0 0x7f8ae0032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 --2- 172.21.15.134:0/1947711422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8af80f62a0 0x7f8af81064b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 >> 172.21.15.134:0/1947711422 conn(0x7f8af80f1aa0 msgr2=0x7f8af80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 shutdown_connections 2023-11-16T01:09:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:00.608+0000 7f8afd577700 1 -- 172.21.15.134:0/1947711422 wait complete. 2023-11-16T01:09:01.012 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T01:09:01.012 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T01:09:01.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:08:58.935946+0000 mgr.smithi133.xdyttl (mgr.14162) 12 : cephadm [INF] Added host smithi134 2023-11-16T01:09:01.890 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:08:59.933438+0000 mgr.smithi133.xdyttl (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:01.891 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:00.606260+0000 mgr.smithi133.xdyttl (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 -- 172.21.15.133:0/1702238354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 msgr2=0x7fdd100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 --2- 172.21.15.133:0/1702238354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd100f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdd04009fb0 tx=0x7fdd04009b30).stop 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 -- 172.21.15.133:0/1702238354 shutdown_connections 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 --2- 172.21.15.133:0/1702238354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd100f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 -- 172.21.15.133:0/1702238354 >> 172.21.15.133:0/1702238354 conn(0x7fdd100f1c70 msgr2=0x7fdd100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 -- 172.21.15.133:0/1702238354 shutdown_connections 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.277+0000 7fdd16f12700 1 -- 172.21.15.133:0/1702238354 wait complete. 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd16f12700 1 Processor -- start 2023-11-16T01:09:02.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd16f12700 1 -- start start 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd16f12700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd16f12700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10106c20 con 0x7fdd100f6510 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd14cae700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd14cae700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47492/0 (socket says 172.21.15.133:47492) 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd14cae700 1 -- 172.21.15.133:0/2943899026 learned_addr learned my addr 172.21.15.133:0/2943899026 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd14cae700 1 -- 172.21.15.133:0/2943899026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04005040 con 0x7fdd100f6510 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd14cae700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdd04014000 tx=0x7fdd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.278+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd040095f0 con 0x7fdd100f6510 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd04007e70 con 0x7fdd100f6510 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd04014ea0 con 0x7fdd100f6510 2023-11-16T01:09:02.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10106e20 con 0x7fdd100f6510 2023-11-16T01:09:02.281 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd101072b0 con 0x7fdd100f6510 2023-11-16T01:09:02.281 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd04025070 con 0x7fdd100f6510 2023-11-16T01:09:02.281 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd0d7fa700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 0x7fdd00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:02.282 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.279+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd04040980 con 0x7fdd100f6510 2023-11-16T01:09:02.282 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.280+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10045ad0 con 0x7fdd100f6510 2023-11-16T01:09:02.282 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.280+0000 7fdd0ffff700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 0x7fdd00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:02.282 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.280+0000 7fdd0ffff700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 0x7fdd00032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fdcfc0096a0 tx=0x7fdcfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:02.284 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.284+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd040079c0 con 0x7fdd100f6510 2023-11-16T01:09:02.439 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.439+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdd100fc050 con 0x7fdd100f6510 2023-11-16T01:09:02.634 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.634+0000 7fdd0d7fa700 1 -- 172.21.15.133:0/2943899026 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdd0402c090 con 0x7fdd100f6510 2023-11-16T01:09:02.636 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 msgr2=0x7fdd00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:02.636 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 0x7fdd00032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fdcfc0096a0 tx=0x7fdcfc006b40).stop 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 msgr2=0x7fdd10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdd04014000 tx=0x7fdd0400b040).stop 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 shutdown_connections 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0002ff60 0x7fdd00032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 --2- 172.21.15.133:0/2943899026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd100f6510 0x7fdd10106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 >> 172.21.15.133:0/2943899026 conn(0x7fdd100f1c70 msgr2=0x7fdd100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 shutdown_connections 2023-11-16T01:09:02.637 INFO:teuthology.orchestra.run.smithi133.stderr:2023-11-16T01:09:02.636+0000 7fdd16f12700 1 -- 172.21.15.133:0/2943899026 wait complete. 2023-11-16T01:09:02.638 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2023-11-16T01:09:02.971 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:02.439795+0000 mon.smithi133 (mon.0) 117 : audit [INF] from='client.? 172.21.15.133:0/2943899026' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T01:09:03.014 INFO:tasks.cephadm:Adding mon.smithi133 on smithi133 2023-11-16T01:09:03.014 INFO:tasks.cephadm:Adding mon.smithi134 on smithi134 2023-11-16T01:09:03.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi133:172.21.15.133=smithi133;smithi134:172.21.15.134=smithi134' 2023-11-16T01:09:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:01.933665+0000 mgr.smithi133.xdyttl (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:02.634704+0000 mon.smithi133 (mon.0) 118 : audit [INF] from='client.? 172.21.15.133:0/2943899026' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T01:09:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:02.634741+0000 mon.smithi133 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-16T01:09:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:03.934012+0000 mgr.smithi133.xdyttl (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:05.934252+0000 mgr.smithi133.xdyttl (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 -- 172.21.15.134:0/2414313501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 msgr2=0x7f69c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 --2- 172.21.15.134:0/2414313501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c40f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f69b8009fb0 tx=0x7f69b8009b30).stop 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 -- 172.21.15.134:0/2414313501 shutdown_connections 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 --2- 172.21.15.134:0/2414313501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c40f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 -- 172.21.15.134:0/2414313501 >> 172.21.15.134:0/2414313501 conn(0x7f69c40f1c70 msgr2=0x7f69c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 -- 172.21.15.134:0/2414313501 shutdown_connections 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.104+0000 7f69c9440700 1 -- 172.21.15.134:0/2414313501 wait complete. 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.105+0000 7f69c9440700 1 Processor -- start 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.105+0000 7f69c9440700 1 -- start start 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c9440700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c9440700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4108de0 con 0x7f69c40f6510 2023-11-16T01:09:09.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36606/0 (socket says 172.21.15.134:36606) 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c2ffd700 1 -- 172.21.15.134:0/3154318144 learned_addr learned my addr 172.21.15.134:0/3154318144 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c2ffd700 1 -- 172.21.15.134:0/3154318144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8005040 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c2ffd700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f69b8013510 tx=0x7f69b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69b8009320 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.106+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c4108fe0 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c4109470 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69b8013f00 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69b8014d60 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69b80263a0 con 0x7f69c40f6510 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69ab7fe700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 0x7f69ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:09.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c4045ad0 con 0x7f69c40f6510 2023-11-16T01:09:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.107+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69b8041320 con 0x7f69c40f6510 2023-11-16T01:09:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.108+0000 7f69c27fc700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 0x7f69ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.108+0000 7f69c27fc700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 0x7f69ac032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f69b40096a0 tx=0x7f69b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.112+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69b800f990 con 0x7f69c40f6510 2023-11-16T01:09:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.287+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 --> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi133:172.21.15.133=smithi133;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f69c40f2e50 con 0x7f69ac02ff60 2023-11-16T01:09:09.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.292+0000 7f69ab7fe700 1 -- 172.21.15.134:0/3154318144 <== mgr.14162 v2:172.21.15.133:6800/2350789018 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f69c40f2e50 con 0x7f69ac02ff60 2023-11-16T01:09:09.293 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2023-11-16T01:09:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 msgr2=0x7f69ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 0x7f69ac032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f69b40096a0 tx=0x7f69b4006b40).stop 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 msgr2=0x7f69c41088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f69b8013510 tx=0x7f69b800b040).stop 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 shutdown_connections 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69ac02ff60 0x7f69ac032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 --2- 172.21.15.134:0/3154318144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c40f6510 0x7f69c41088f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 >> 172.21.15.134:0/3154318144 conn(0x7f69c40f1c70 msgr2=0x7f69c40f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 shutdown_connections 2023-11-16T01:09:09.295 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:09.294+0000 7f69c9440700 1 -- 172.21.15.134:0/3154318144 wait complete. 2023-11-16T01:09:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:07.934487+0000 mgr.smithi133.xdyttl (mgr.14162) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:09.292598+0000 mon.smithi133 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:09:10.359 DEBUG:teuthology.orchestra.run.smithi134:mon.smithi134> sudo journalctl -f -n 0 -u ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134.service 2023-11-16T01:09:10.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:10.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:10.392 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:-- Logs begin at Thu 2023-11-16 00:51:20 UTC. -- 2023-11-16T01:09:11.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:09.288688+0000 mgr.smithi133.xdyttl (mgr.14162) 19 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi133:172.21.15.133=smithi133;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T01:09:11.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:09:09.290086+0000 mgr.smithi133.xdyttl (mgr.14162) 20 : cephadm [INF] Saving service mon spec with placement smithi133:172.21.15.133=smithi133;smithi134:172.21.15.134=smithi134;count:2 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- 172.21.15.134:0/2279717775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 msgr2=0x7fe2480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2279717775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe2480f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe244009fb0 tx=0x7fe244009b30).stop 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- 172.21.15.134:0/2279717775 shutdown_connections 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2279717775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe2480f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- 172.21.15.134:0/2279717775 >> 172.21.15.134:0/2279717775 conn(0x7fe2480f0b70 msgr2=0x7fe2480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- 172.21.15.134:0/2279717775 shutdown_connections 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- 172.21.15.134:0/2279717775 wait complete. 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 Processor -- start 2023-11-16T01:09:11.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.556+0000 7fe2506b3700 1 -- start start 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe2506b3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe2506b3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe248106c10 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe24e44f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe24e44f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58802/0 (socket says 172.21.15.134:58802) 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe24e44f700 1 -- 172.21.15.134:0/2966552433 learned_addr learned my addr 172.21.15.134:0/2966552433 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe24e44f700 1 -- 172.21.15.134:0/2966552433 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244005040 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe24e44f700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe244013510 tx=0x7fe24400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe244009320 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe248106e10 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe244013f00 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe244014d90 con 0x7fe2480f5410 2023-11-16T01:09:11.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.557+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2481072a0 con 0x7fe2480f5410 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.558+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2440095f0 con 0x7fe2480f5410 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.558+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe248045ad0 con 0x7fe2480f5410 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.558+0000 7fe23effd700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 0x7fe234032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.558+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe244040fe0 con 0x7fe2480f5410 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.559+0000 7fe24dc4e700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 0x7fe234032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.559+0000 7fe24dc4e700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 0x7fe234032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe2380096a0 tx=0x7fe238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:11.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.563+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe24401d7f0 con 0x7fe2480f5410 2023-11-16T01:09:11.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.766+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe248108150 con 0x7fe2480f5410 2023-11-16T01:09:11.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.767+0000 7fe23effd700 1 -- 172.21.15.134:0/2966552433 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe244010230 con 0x7fe2480f5410 2023-11-16T01:09:11.767 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:11.767 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 msgr2=0x7fe234032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 0x7fe234032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe2380096a0 tx=0x7fe238006b40).stop 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 msgr2=0x7fe248106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe244013510 tx=0x7fe24400b040).stop 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 shutdown_connections 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe23402ff60 0x7fe234032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 --2- 172.21.15.134:0/2966552433 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2480f5410 0x7fe248106720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:11.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 >> 172.21.15.134:0/2966552433 conn(0x7fe2480f0b70 msgr2=0x7fe24805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 shutdown_connections 2023-11-16T01:09:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:11.768+0000 7fe2506b3700 1 -- 172.21.15.134:0/2966552433 wait complete. 2023-11-16T01:09:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:09.934723+0000 mgr.smithi133.xdyttl (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:13.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:12 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:11.767164+0000 mon.smithi133 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.134:0/2966552433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:13.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:13.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:11.935015+0000 mgr.smithi133.xdyttl (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:14.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.245+0000 7f1d82e63700 1 -- 172.21.15.134:0/373574893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 msgr2=0x7f1d7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 --2- 172.21.15.134:0/373574893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c0f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1d70009fb0 tx=0x7f1d70009b30).stop 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- 172.21.15.134:0/373574893 shutdown_connections 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 --2- 172.21.15.134:0/373574893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c0f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- 172.21.15.134:0/373574893 >> 172.21.15.134:0/373574893 conn(0x7f1d7c0f0b70 msgr2=0x7f1d7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- 172.21.15.134:0/373574893 shutdown_connections 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- 172.21.15.134:0/373574893 wait complete. 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 Processor -- start 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- start start 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:14.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.246+0000 7f1d82e63700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1071e0 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d80bff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d80bff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58826/0 (socket says 172.21.15.134:58826) 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d80bff700 1 -- 172.21.15.134:0/1102847297 learned_addr learned my addr 172.21.15.134:0/1102847297 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d80bff700 1 -- 172.21.15.134:0/1102847297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70005040 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d80bff700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1d70013510 tx=0x7f1d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d70009320 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c060220 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d70013f00 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d70014d90 con 0x7f1d7c0f5410 2023-11-16T01:09:14.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.247+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c10bce0 con 0x7f1d7c0f5410 2023-11-16T01:09:14.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.248+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d700263a0 con 0x7f1d7c0f5410 2023-11-16T01:09:14.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.248+0000 7f1d797fa700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:14.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.248+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d70041060 con 0x7f1d7c0f5410 2023-11-16T01:09:14.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.248+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c059100 con 0x7f1d7c0f5410 2023-11-16T01:09:14.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.248+0000 7f1d7bfff700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:14.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.249+0000 7f1d7bfff700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 0x7f1d6c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1d680096a0 tx=0x7f1d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:14.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.253+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d70012920 con 0x7f1d7c0f5410 2023-11-16T01:09:14.462 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.460+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d7c045ad0 con 0x7f1d7c0f5410 2023-11-16T01:09:14.462 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.461+0000 7f1d797fa700 1 -- 172.21.15.134:0/1102847297 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d7002b020 con 0x7f1d7c0f5410 2023-11-16T01:09:14.463 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:14.463 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:14.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 msgr2=0x7f1d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:14.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 0x7f1d6c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1d680096a0 tx=0x7f1d68006b40).stop 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 msgr2=0x7f1d7c10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1d70013510 tx=0x7f1d7000b040).stop 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 shutdown_connections 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 --2- 172.21.15.134:0/1102847297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d7c0f5410 0x7f1d7c10b7f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 >> 172.21.15.134:0/1102847297 conn(0x7f1d7c0f0b70 msgr2=0x7f1d7c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 shutdown_connections 2023-11-16T01:09:14.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:14.463+0000 7f1d82e63700 1 -- 172.21.15.134:0/1102847297 wait complete. 2023-11-16T01:09:14.466 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:15.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:14 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:14.462823+0000 mon.smithi133 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.134:0/1102847297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:15.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:15.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:13.935350+0000 mgr.smithi133.xdyttl (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:17.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 -- 172.21.15.134:0/3014734977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 msgr2=0x7f7d400f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:17.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 --2- 172.21.15.134:0/3014734977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d400f4c80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7d30009fb0 tx=0x7f7d30009b30).stop 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 -- 172.21.15.134:0/3014734977 shutdown_connections 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 --2- 172.21.15.134:0/3014734977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d400f4c80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 -- 172.21.15.134:0/3014734977 >> 172.21.15.134:0/3014734977 conn(0x7f7d400f02e0 msgr2=0x7f7d400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 -- 172.21.15.134:0/3014734977 shutdown_connections 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.105+0000 7f7d46708700 1 -- 172.21.15.134:0/3014734977 wait complete. 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d46708700 1 Processor -- start 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d46708700 1 -- start start 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d46708700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d46708700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40106dd0 con 0x7f7d400f4860 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58846/0 (socket says 172.21.15.134:58846) 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.106+0000 7f7d3ffff700 1 -- 172.21.15.134:0/2780220933 learned_addr learned my addr 172.21.15.134:0/2780220933 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d3ffff700 1 -- 172.21.15.134:0/2780220933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30005040 con 0x7f7d400f4860 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d3ffff700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7d30013510 tx=0x7f7d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d30009320 con 0x7f7d400f4860 2023-11-16T01:09:17.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40106fd0 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d30013f00 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d30014dc0 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.107+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d40107460 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d30014420 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40045ad0 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d3cff9700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d300410b0 con 0x7f7d400f4860 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d3f7fe700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:17.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.108+0000 7f7d3f7fe700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 0x7f7d28032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7d340096a0 tx=0x7f7d34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:17.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.112+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d300095f0 con 0x7f7d400f4860 2023-11-16T01:09:17.317 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.316+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d4005f370 con 0x7f7d400f4860 2023-11-16T01:09:17.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.317+0000 7f7d3cff9700 1 -- 172.21.15.134:0/2780220933 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d3001cb10 con 0x7f7d400f4860 2023-11-16T01:09:17.320 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:17.320 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:17.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.318+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 msgr2=0x7f7d28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.318+0000 7f7d46708700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 0x7f7d28032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7d340096a0 tx=0x7f7d34006b40).stop 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.318+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 msgr2=0x7f7d401068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.318+0000 7f7d46708700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7d30013510 tx=0x7f7d3000b040).stop 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 shutdown_connections 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 --2- 172.21.15.134:0/2780220933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d400f4860 0x7f7d401068e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 >> 172.21.15.134:0/2780220933 conn(0x7f7d400f02e0 msgr2=0x7f7d400f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 shutdown_connections 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:17.319+0000 7f7d46708700 1 -- 172.21.15.134:0/2780220933 wait complete. 2023-11-16T01:09:17.321 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:15.935676+0000 mgr.smithi133.xdyttl (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:17.318350+0000 mon.smithi133 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.134:0/2780220933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:18.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:18.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 -- 172.21.15.134:0/119609162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 msgr2=0x7f920c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 --2- 172.21.15.134:0/119609162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c0f9150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9200009fb0 tx=0x7f9200009b30).stop 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 -- 172.21.15.134:0/119609162 shutdown_connections 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 --2- 172.21.15.134:0/119609162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c0f9150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 -- 172.21.15.134:0/119609162 >> 172.21.15.134:0/119609162 conn(0x7f920c0f0b70 msgr2=0x7f920c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:19.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 -- 172.21.15.134:0/119609162 shutdown_connections 2023-11-16T01:09:19.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 -- 172.21.15.134:0/119609162 wait complete. 2023-11-16T01:09:19.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.877+0000 7f9212a4d700 1 Processor -- start 2023-11-16T01:09:19.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f9212a4d700 1 -- start start 2023-11-16T01:09:19.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f9212a4d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:19.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f9212a4d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c106bd0 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f920bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f920bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:44558/0 (socket says 172.21.15.134:44558) 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f920bfff700 1 -- 172.21.15.134:0/3462734310 learned_addr learned my addr 172.21.15.134:0/3462734310 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f920bfff700 1 -- 172.21.15.134:0/3462734310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200005040 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.878+0000 7f920bfff700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9200013510 tx=0x7f920000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9200009320 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c106dd0 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9200013f00 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9200014dc0 con 0x7f920c060220 2023-11-16T01:09:19.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c107260 con 0x7f920c060220 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92000095f0 con 0x7f920c060220 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.879+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920c045ad0 con 0x7f920c060220 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.880+0000 7f9208ff9700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 0x7f91f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.880+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9200041040 con 0x7f920c060220 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.880+0000 7f920b7fe700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 0x7f91f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.880+0000 7f920b7fe700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 0x7f91f4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f91fc0096a0 tx=0x7f91fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:19.885 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:19.884+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9200014420 con 0x7f920c060220 2023-11-16T01:09:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:17.935976+0000 mgr.smithi133.xdyttl (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:20.094 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.092+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f920c108110 con 0x7f920c060220 2023-11-16T01:09:20.094 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.093+0000 7f9208ff9700 1 -- 172.21.15.134:0/3462734310 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9200010230 con 0x7f920c060220 2023-11-16T01:09:20.095 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:20.095 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.094+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 msgr2=0x7f91f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.094+0000 7f9212a4d700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 0x7f91f4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f91fc0096a0 tx=0x7f91fc006b40).stop 2023-11-16T01:09:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.094+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 msgr2=0x7f920c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.094+0000 7f9212a4d700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9200013510 tx=0x7f920000b040).stop 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 shutdown_connections 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f91f402ff60 0x7f91f4032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 --2- 172.21.15.134:0/3462734310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f920c060220 0x7f920c1066e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 >> 172.21.15.134:0/3462734310 conn(0x7f920c0f0b70 msgr2=0x7f920c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 shutdown_connections 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:20.095+0000 7f9212a4d700 1 -- 172.21.15.134:0/3462734310 wait complete. 2023-11-16T01:09:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:21.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:20.094486+0000 mon.smithi133 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.134:0/3462734310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:21.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:21.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:22.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:19.936307+0000 mgr.smithi133.xdyttl (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:22.632 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 -- 172.21.15.134:0/2267098738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 msgr2=0x7fce3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:22.632 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 --2- 172.21.15.134:0/2267098738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fce24009fb0 tx=0x7fce24009b30).stop 2023-11-16T01:09:22.632 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 -- 172.21.15.134:0/2267098738 shutdown_connections 2023-11-16T01:09:22.632 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 --2- 172.21.15.134:0/2267098738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:22.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 -- 172.21.15.134:0/2267098738 >> 172.21.15.134:0/2267098738 conn(0x7fce3c0f0b70 msgr2=0x7fce3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:22.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 -- 172.21.15.134:0/2267098738 shutdown_connections 2023-11-16T01:09:22.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.630+0000 7fce423e2700 1 -- 172.21.15.134:0/2267098738 wait complete. 2023-11-16T01:09:22.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce423e2700 1 Processor -- start 2023-11-16T01:09:22.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce423e2700 1 -- start start 2023-11-16T01:09:22.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce423e2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce423e2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c060220 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce3bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce3bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:44586/0 (socket says 172.21.15.134:44586) 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.631+0000 7fce3bfff700 1 -- 172.21.15.134:0/182978428 learned_addr learned my addr 172.21.15.134:0/182978428 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce3bfff700 1 -- 172.21.15.134:0/182978428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24005040 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce3bfff700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fce24013510 tx=0x7fce2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce24009320 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c060420 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce24013f00 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce24014d90 con 0x7fce3c0f5410 2023-11-16T01:09:22.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.632+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c109b00 con 0x7fce3c0f5410 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.633+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce240095f0 con 0x7fce3c0f5410 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.633+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c059100 con 0x7fce3c0f5410 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.633+0000 7fce38ff9700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 0x7fce28032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.633+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce24040fe0 con 0x7fce3c0f5410 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.633+0000 7fce3b7fe700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 0x7fce28032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:22.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.634+0000 7fce3b7fe700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 0x7fce28032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fce2c0096a0 tx=0x7fce2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:22.639 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.638+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce2406d2b0 con 0x7fce3c0f5410 2023-11-16T01:09:22.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.845+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce3c045ad0 con 0x7fce3c0f5410 2023-11-16T01:09:22.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.845+0000 7fce38ff9700 1 -- 172.21.15.134:0/182978428 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce24014420 con 0x7fce3c0f5410 2023-11-16T01:09:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:22.848 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:22.849 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 msgr2=0x7fce28032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:22.849 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 0x7fce28032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fce2c0096a0 tx=0x7fce2c006b40).stop 2023-11-16T01:09:22.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 msgr2=0x7fce3c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:22.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fce24013510 tx=0x7fce2400b040).stop 2023-11-16T01:09:22.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 shutdown_connections 2023-11-16T01:09:22.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce2802ffb0 0x7fce28032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:22.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 --2- 172.21.15.134:0/182978428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce3c0f5410 0x7fce3c109450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 >> 172.21.15.134:0/182978428 conn(0x7fce3c0f0b70 msgr2=0x7fce3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 shutdown_connections 2023-11-16T01:09:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:22.847+0000 7fce423e2700 1 -- 172.21.15.134:0/182978428 wait complete. 2023-11-16T01:09:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:21.936702+0000 mgr.smithi133.xdyttl (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:22.846869+0000 mon.smithi133 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.134:0/182978428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:24.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:24.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 -- 172.21.15.134:0/2793488358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 msgr2=0x7fd4e40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 --2- 172.21.15.134:0/2793488358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e40f4be0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd4d8009fb0 tx=0x7fd4d8009b30).stop 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 -- 172.21.15.134:0/2793488358 shutdown_connections 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 --2- 172.21.15.134:0/2793488358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e40f4be0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 -- 172.21.15.134:0/2793488358 >> 172.21.15.134:0/2793488358 conn(0x7fd4e40f02e0 msgr2=0x7fd4e40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 -- 172.21.15.134:0/2793488358 shutdown_connections 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.350+0000 7fd4ec176700 1 -- 172.21.15.134:0/2793488358 wait complete. 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4ec176700 1 Processor -- start 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4ec176700 1 -- start start 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4ec176700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4ec176700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e4106bf0 con 0x7fd4e40f47c0 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4e9f12700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4e9f12700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:44594/0 (socket says 172.21.15.134:44594) 2023-11-16T01:09:25.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4e9f12700 1 -- 172.21.15.134:0/1037247027 learned_addr learned my addr 172.21.15.134:0/1037247027 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:25.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4e9f12700 1 -- 172.21.15.134:0/1037247027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8005040 con 0x7fd4e40f47c0 2023-11-16T01:09:25.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.351+0000 7fd4e9f12700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd4d8013510 tx=0x7fd4d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:25.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d8009320 con 0x7fd4e40f47c0 2023-11-16T01:09:25.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4d8013f00 con 0x7fd4e40f47c0 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e4106df0 con 0x7fd4e40f47c0 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d8014d90 con 0x7fd4e40f47c0 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e4107280 con 0x7fd4e40f47c0 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4d8014420 con 0x7fd4e40f47c0 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4d67fc700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 0x7fd4d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:25.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.352+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e4045ad0 con 0x7fd4e40f47c0 2023-11-16T01:09:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.353+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4d8041000 con 0x7fd4e40f47c0 2023-11-16T01:09:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.353+0000 7fd4e9711700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 0x7fd4d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.353+0000 7fd4e9711700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 0x7fd4d0032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd4e00096a0 tx=0x7fd4e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.357+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4d80095f0 con 0x7fd4e40f47c0 2023-11-16T01:09:25.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.566+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4e4108130 con 0x7fd4e40f47c0 2023-11-16T01:09:25.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.567+0000 7fd4d67fc700 1 -- 172.21.15.134:0/1037247027 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4d802c090 con 0x7fd4e40f47c0 2023-11-16T01:09:25.569 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:25.569 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:25.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 msgr2=0x7fd4d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:25.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 0x7fd4d0032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd4e00096a0 tx=0x7fd4e0006b40).stop 2023-11-16T01:09:25.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 msgr2=0x7fd4e4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd4d8013510 tx=0x7fd4d800b040).stop 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 shutdown_connections 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd4d002ff60 0x7fd4d0032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 --2- 172.21.15.134:0/1037247027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4e40f47c0 0x7fd4e4106700 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 >> 172.21.15.134:0/1037247027 conn(0x7fd4e40f02e0 msgr2=0x7fd4e40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 shutdown_connections 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:25.569+0000 7fd4ec176700 1 -- 172.21.15.134:0/1037247027 wait complete. 2023-11-16T01:09:25.571 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:23.936998+0000 mgr.smithi133.xdyttl (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:25.568578+0000 mon.smithi133 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.134:0/1037247027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:25.937357+0000 mgr.smithi133.xdyttl (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:32.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:32.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:32.685 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:27.937735+0000 mgr.smithi133.xdyttl (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:32.685 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:30.070849+0000 mon.smithi133 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:09:32.685 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:30.072520+0000 mon.smithi133 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:09:32.685 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:29.938012+0000 mgr.smithi133.xdyttl (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:33.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 -- 172.21.15.134:0/1454722740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 msgr2=0x7f62240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:33.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 --2- 172.21.15.134:0/1454722740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f62240f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f620c009fb0 tx=0x7f620c009b30).stop 2023-11-16T01:09:33.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 -- 172.21.15.134:0/1454722740 shutdown_connections 2023-11-16T01:09:33.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 --2- 172.21.15.134:0/1454722740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f62240f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:33.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 -- 172.21.15.134:0/1454722740 >> 172.21.15.134:0/1454722740 conn(0x7f62240f0b70 msgr2=0x7f62240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:33.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 -- 172.21.15.134:0/1454722740 shutdown_connections 2023-11-16T01:09:33.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.817+0000 7f62299ad700 1 -- 172.21.15.134:0/1454722740 wait complete. 2023-11-16T01:09:33.821 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.818+0000 7f62299ad700 1 Processor -- start 2023-11-16T01:09:33.821 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.818+0000 7f62299ad700 1 -- start start 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.818+0000 7f62299ad700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.818+0000 7f62299ad700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224105b20 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60328/0 (socket says 172.21.15.134:60328) 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f6222ffd700 1 -- 172.21.15.134:0/324121667 learned_addr learned my addr 172.21.15.134:0/324121667 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f6222ffd700 1 -- 172.21.15.134:0/324121667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c005040 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f6222ffd700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f620c013510 tx=0x7f620c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f620c009320 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6224105d20 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f620c013f00 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f620c014d90 con 0x7f62240f5410 2023-11-16T01:09:33.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.819+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62241061b0 con 0x7f62240f5410 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.820+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f620c0095f0 con 0x7f62240f5410 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.820+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6224045ad0 con 0x7f62240f5410 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.820+0000 7f621b7fe700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 0x7f6210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.820+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f620c0410c0 con 0x7f62240f5410 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.820+0000 7f62227fc700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 0x7f6210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.821+0000 7f62227fc700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 0x7f6210032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f62140096a0 tx=0x7f6214006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:33.826 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:33.825+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f620c014420 con 0x7f62240f5410 2023-11-16T01:09:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:31.938386+0000 mgr.smithi133.xdyttl (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:34.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.029+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62240fae60 con 0x7f62240f5410 2023-11-16T01:09:34.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.030+0000 7f621b7fe700 1 -- 172.21.15.134:0/324121667 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f620c01d7f0 con 0x7f62240f5410 2023-11-16T01:09:34.032 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:34.032 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:34.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.031+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 msgr2=0x7f6210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:34.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.031+0000 7f62299ad700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 0x7f6210032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f62140096a0 tx=0x7f6214006b40).stop 2023-11-16T01:09:34.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.031+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 msgr2=0x7f6224105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:34.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.031+0000 7f62299ad700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f620c013510 tx=0x7f620c00b040).stop 2023-11-16T01:09:34.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 shutdown_connections 2023-11-16T01:09:34.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f621002ff60 0x7f6210032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:34.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 --2- 172.21.15.134:0/324121667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62240f5410 0x7f6224105630 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:34.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 >> 172.21.15.134:0/324121667 conn(0x7f62240f0b70 msgr2=0x7f62240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:34.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 shutdown_connections 2023-11-16T01:09:34.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:34.032+0000 7f62299ad700 1 -- 172.21.15.134:0/324121667 wait complete. 2023-11-16T01:09:34.036 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:35.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:34 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:34.031614+0000 mon.smithi133 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.134:0/324121667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:35.434 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:35.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:33.938703+0000 mgr.smithi133.xdyttl (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:36.614 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 -- 172.21.15.134:0/501502964 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 msgr2=0x7f773c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 --2- 172.21.15.134:0/501502964 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c0f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7730009fb0 tx=0x7f7730009b30).stop 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 -- 172.21.15.134:0/501502964 shutdown_connections 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 --2- 172.21.15.134:0/501502964 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c0f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 -- 172.21.15.134:0/501502964 >> 172.21.15.134:0/501502964 conn(0x7f773c0f0b70 msgr2=0x7f773c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 -- 172.21.15.134:0/501502964 shutdown_connections 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.611+0000 7f7743aba700 1 -- 172.21.15.134:0/501502964 wait complete. 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7743aba700 1 Processor -- start 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7743aba700 1 -- start start 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7743aba700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7743aba700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c060220 con 0x7f773c0f5410 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7741856700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:36.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7741856700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60354/0 (socket says 172.21.15.134:60354) 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.612+0000 7f7741856700 1 -- 172.21.15.134:0/1969151329 learned_addr learned my addr 172.21.15.134:0/1969151329 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f7741856700 1 -- 172.21.15.134:0/1969151329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7730005040 con 0x7f773c0f5410 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f7741856700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7730013510 tx=0x7f773000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7730009320 con 0x7f773c0f5410 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c060420 con 0x7f773c0f5410 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7730013f00 con 0x7f773c0f5410 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7730014d90 con 0x7f773c0f5410 2023-11-16T01:09:36.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.613+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c109b00 con 0x7f773c0f5410 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.614+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77300263a0 con 0x7f773c0f5410 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.614+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c059100 con 0x7f773c0f5410 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.614+0000 7f772e7fc700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.614+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7730041070 con 0x7f773c0f5410 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.614+0000 7f7741055700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:36.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.615+0000 7f7741055700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 0x7f7728032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f77380096a0 tx=0x7f7738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:36.620 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.619+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77300126f0 con 0x7f773c0f5410 2023-11-16T01:09:36.826 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.825+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f773c045ad0 con 0x7f773c0f5410 2023-11-16T01:09:36.827 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.825+0000 7f772e7fc700 1 -- 172.21.15.134:0/1969151329 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7730012330 con 0x7f773c0f5410 2023-11-16T01:09:36.827 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:36.827 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 msgr2=0x7f7728032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 0x7f7728032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f77380096a0 tx=0x7f7738006b40).stop 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 msgr2=0x7f773c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7730013510 tx=0x7f773000b040).stop 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 shutdown_connections 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 --2- 172.21.15.134:0/1969151329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f773c0f5410 0x7f773c109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 >> 172.21.15.134:0/1969151329 conn(0x7f773c0f0b70 msgr2=0x7f773c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 shutdown_connections 2023-11-16T01:09:36.829 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:36.827+0000 7f7743aba700 1 -- 172.21.15.134:0/1969151329 wait complete. 2023-11-16T01:09:36.830 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:38.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:35.938986+0000 mgr.smithi133.xdyttl (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:38.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:36.827022+0000 mon.smithi133 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.134:0/1969151329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:38.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:38.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:39.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 -- 172.21.15.134:0/1283628336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 msgr2=0x7f67b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:39.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 --2- 172.21.15.134:0/1283628336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b80f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f67b4009fb0 tx=0x7f67b4009b30).stop 2023-11-16T01:09:39.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 -- 172.21.15.134:0/1283628336 shutdown_connections 2023-11-16T01:09:39.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 --2- 172.21.15.134:0/1283628336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b80f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 -- 172.21.15.134:0/1283628336 >> 172.21.15.134:0/1283628336 conn(0x7f67b80f1c70 msgr2=0x7f67b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 -- 172.21.15.134:0/1283628336 shutdown_connections 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.429+0000 7f67bf638700 1 -- 172.21.15.134:0/1283628336 wait complete. 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bf638700 1 Processor -- start 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bf638700 1 -- start start 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bf638700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:39.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bf638700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b8106c20 con 0x7f67b80f6510 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bd3d4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bd3d4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49028/0 (socket says 172.21.15.134:49028) 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bd3d4700 1 -- 172.21.15.134:0/145759457 learned_addr learned my addr 172.21.15.134:0/145759457 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.430+0000 7f67bd3d4700 1 -- 172.21.15.134:0/145759457 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b4005040 con 0x7f67b80f6510 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67bd3d4700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f67b4013510 tx=0x7f67b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b4009320 con 0x7f67b80f6510 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b8106e20 con 0x7f67b80f6510 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67b4013f00 con 0x7f67b80f6510 2023-11-16T01:09:39.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b4014d90 con 0x7f67b80f6510 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.431+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b81072b0 con 0x7f67b80f6510 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.432+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67b4014420 con 0x7f67b80f6510 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.432+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b8045ad0 con 0x7f67b80f6510 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.432+0000 7f67adffb700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.432+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67b4041110 con 0x7f67b80f6510 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.432+0000 7f67bcbd3700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.433+0000 7f67bcbd3700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 0x7f67a4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f67a80096a0 tx=0x7f67a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:39.437 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.437+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67b40095f0 con 0x7f67b80f6510 2023-11-16T01:09:39.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.641+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67b80fbf60 con 0x7f67b80f6510 2023-11-16T01:09:39.642 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.641+0000 7f67adffb700 1 -- 172.21.15.134:0/145759457 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67b401db70 con 0x7f67b80f6510 2023-11-16T01:09:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:39.642 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:39.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 msgr2=0x7f67a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 0x7f67a4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f67a80096a0 tx=0x7f67a8006b40).stop 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 msgr2=0x7f67b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f67b4013510 tx=0x7f67b400b040).stop 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 shutdown_connections 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ff60 0x7f67a4032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 --2- 172.21.15.134:0/145759457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67b80f6510 0x7f67b8106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 >> 172.21.15.134:0/145759457 conn(0x7f67b80f1c70 msgr2=0x7f67b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 shutdown_connections 2023-11-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:39.643+0000 7f67bf638700 1 -- 172.21.15.134:0/145759457 wait complete. 2023-11-16T01:09:39.645 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:37.939242+0000 mgr.smithi133.xdyttl (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:39.642016+0000 mon.smithi133 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.134:0/145759457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:41.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:41.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:42.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:39.939510+0000 mgr.smithi133.xdyttl (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 -- 172.21.15.134:0/2275516804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 msgr2=0x7fd9640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 --2- 172.21.15.134:0/2275516804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9640f90f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd958009fb0 tx=0x7fd958009b30).stop 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 -- 172.21.15.134:0/2275516804 shutdown_connections 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 --2- 172.21.15.134:0/2275516804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9640f90f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 -- 172.21.15.134:0/2275516804 >> 172.21.15.134:0/2275516804 conn(0x7fd9640f0b70 msgr2=0x7fd9640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 -- 172.21.15.134:0/2275516804 shutdown_connections 2023-11-16T01:09:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.176+0000 7fd968e54700 1 -- 172.21.15.134:0/2275516804 wait complete. 2023-11-16T01:09:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd968e54700 1 Processor -- start 2023-11-16T01:09:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd968e54700 1 -- start start 2023-11-16T01:09:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd968e54700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd968e54700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd964106bd0 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd96259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd96259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49052/0 (socket says 172.21.15.134:49052) 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd96259c700 1 -- 172.21.15.134:0/3821164559 learned_addr learned my addr 172.21.15.134:0/3821164559 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.177+0000 7fd96259c700 1 -- 172.21.15.134:0/3821164559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958005040 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd96259c700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd958013510 tx=0x7fd95800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd958009320 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd964106dd0 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd958013f00 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd958014dc0 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.178+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd964107260 con 0x7fd964060280 2023-11-16T01:09:42.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9580095f0 con 0x7fd964060280 2023-11-16T01:09:42.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964045ad0 con 0x7fd964060280 2023-11-16T01:09:42.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd952ffd700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 0x7fd94c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:42.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9580410c0 con 0x7fd964060280 2023-11-16T01:09:42.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd961d9b700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 0x7fd94c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:42.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.179+0000 7fd961d9b700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 0x7fd94c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd9540096a0 tx=0x7fd954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:42.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.183+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd958014420 con 0x7fd964060280 2023-11-16T01:09:42.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.390+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd964108110 con 0x7fd964060280 2023-11-16T01:09:42.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.391+0000 7fd952ffd700 1 -- 172.21.15.134:0/3821164559 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd958010230 con 0x7fd964060280 2023-11-16T01:09:42.392 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:42.392 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:42.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 msgr2=0x7fd94c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:42.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 0x7fd94c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd9540096a0 tx=0x7fd954006b40).stop 2023-11-16T01:09:42.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 msgr2=0x7fd9641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:42.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd958013510 tx=0x7fd95800b040).stop 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 shutdown_connections 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd94c02ff60 0x7fd94c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 --2- 172.21.15.134:0/3821164559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd964060280 0x7fd9641066e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 >> 172.21.15.134:0/3821164559 conn(0x7fd9640f0b70 msgr2=0x7fd9640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 shutdown_connections 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:42.393+0000 7fd968e54700 1 -- 172.21.15.134:0/3821164559 wait complete. 2023-11-16T01:09:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:43.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:42 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:42.391700+0000 mon.smithi133 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.134:0/3821164559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:43.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:43.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:44.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:41.939850+0000 mgr.smithi133.xdyttl (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 -- 172.21.15.134:0/4173821625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 msgr2=0x7f58840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 --2- 172.21.15.134:0/4173821625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f58840f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5874009fb0 tx=0x7f5874009b30).stop 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 -- 172.21.15.134:0/4173821625 shutdown_connections 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 --2- 172.21.15.134:0/4173821625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f58840f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 -- 172.21.15.134:0/4173821625 >> 172.21.15.134:0/4173821625 conn(0x7f58840f1c70 msgr2=0x7f58840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 -- 172.21.15.134:0/4173821625 shutdown_connections 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.932+0000 7f588b467700 1 -- 172.21.15.134:0/4173821625 wait complete. 2023-11-16T01:09:44.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.933+0000 7f588b467700 1 Processor -- start 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.933+0000 7f588b467700 1 -- start start 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.933+0000 7f588b467700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.933+0000 7f588b467700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5884106c20 con 0x7f58840f6510 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5889203700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5889203700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49074/0 (socket says 172.21.15.134:49074) 2023-11-16T01:09:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5889203700 1 -- 172.21.15.134:0/973851404 learned_addr learned my addr 172.21.15.134:0/973851404 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5889203700 1 -- 172.21.15.134:0/973851404 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874005040 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5889203700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5874013510 tx=0x7f587400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874009320 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5884106e20 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5874013f00 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874014d90 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.934+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58841072b0 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.935+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58740263a0 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.935+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5884045ad0 con 0x7f58840f6510 2023-11-16T01:09:44.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.935+0000 7f5879ffb700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:44.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.935+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5874041390 con 0x7f58840f6510 2023-11-16T01:09:44.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.935+0000 7f5888a02700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:44.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.936+0000 7f5888a02700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 0x7f5870032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f58800096a0 tx=0x7f5880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:44.940 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:44.940+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58740095f0 con 0x7f58840f6510 2023-11-16T01:09:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.145+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58840fbf60 con 0x7f58840f6510 2023-11-16T01:09:45.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.146+0000 7f5879ffb700 1 -- 172.21.15.134:0/973851404 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f58740095f0 con 0x7f58840f6510 2023-11-16T01:09:45.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:45.147 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 msgr2=0x7f5870032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 0x7f5870032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f58800096a0 tx=0x7f5880006b40).stop 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 msgr2=0x7f5884106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5874013510 tx=0x7f587400b040).stop 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 shutdown_connections 2023-11-16T01:09:45.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.147+0000 7f588b467700 1 --2- 172.21.15.134:0/973851404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58840f6510 0x7f5884106730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.148+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 >> 172.21.15.134:0/973851404 conn(0x7f58840f1c70 msgr2=0x7f58840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.148+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 shutdown_connections 2023-11-16T01:09:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:45.148+0000 7f588b467700 1 -- 172.21.15.134:0/973851404 wait complete. 2023-11-16T01:09:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:46.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:43.940124+0000 mgr.smithi133.xdyttl (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:46.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:45.146424+0000 mon.smithi133 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.134:0/973851404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:46.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:46.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:47.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 -- 172.21.15.134:0/1613958510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 msgr2=0x7f8d240f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:47.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 --2- 172.21.15.134:0/1613958510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d240f4be0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8d18009fb0 tx=0x7f8d18009b30).stop 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 -- 172.21.15.134:0/1613958510 shutdown_connections 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 --2- 172.21.15.134:0/1613958510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d240f4be0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 -- 172.21.15.134:0/1613958510 >> 172.21.15.134:0/1613958510 conn(0x7f8d240f02e0 msgr2=0x7f8d240f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 -- 172.21.15.134:0/1613958510 shutdown_connections 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.615+0000 7f8d29a45700 1 -- 172.21.15.134:0/1613958510 wait complete. 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d29a45700 1 Processor -- start 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d29a45700 1 -- start start 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d29a45700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:47.617 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d29a45700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24106bf0 con 0x7f8d240f47c0 2023-11-16T01:09:47.618 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d22ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d22ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49090/0 (socket says 172.21.15.134:49090) 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.616+0000 7f8d22ffd700 1 -- 172.21.15.134:0/2021190641 learned_addr learned my addr 172.21.15.134:0/2021190641 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d22ffd700 1 -- 172.21.15.134:0/2021190641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18005040 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d22ffd700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8d18013510 tx=0x7f8d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d18009320 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d18013f00 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d18014d90 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d24106df0 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.617+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d24107280 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.618+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d18014420 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.618+0000 7f8d0b7fe700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 0x7f8d0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.618+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d18040fe0 con 0x7f8d240f47c0 2023-11-16T01:09:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.618+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24045ad0 con 0x7f8d240f47c0 2023-11-16T01:09:47.620 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.618+0000 7f8d227fc700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 0x7f8d0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:47.620 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.619+0000 7f8d227fc700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 0x7f8d0c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8d140096a0 tx=0x7f8d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:47.623 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.623+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d180095f0 con 0x7f8d240f47c0 2023-11-16T01:09:47.834 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.834+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d24108130 con 0x7f8d240f47c0 2023-11-16T01:09:47.835 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.835+0000 7f8d0b7fe700 1 -- 172.21.15.134:0/2021190641 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d1801db70 con 0x7f8d240f47c0 2023-11-16T01:09:47.835 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:47.835 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:47.837 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.836+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 msgr2=0x7f8d0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:47.837 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.836+0000 7f8d29a45700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 0x7f8d0c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8d140096a0 tx=0x7f8d14006b40).stop 2023-11-16T01:09:47.837 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 msgr2=0x7f8d24106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8d18013510 tx=0x7f8d1800b040).stop 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 shutdown_connections 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8d0c02ff60 0x7f8d0c032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 --2- 172.21.15.134:0/2021190641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d240f47c0 0x7f8d24106700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 >> 172.21.15.134:0/2021190641 conn(0x7f8d240f02e0 msgr2=0x7f8d240f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 shutdown_connections 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:47.837+0000 7f8d29a45700 1 -- 172.21.15.134:0/2021190641 wait complete. 2023-11-16T01:09:47.838 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:45.940385+0000 mgr.smithi133.xdyttl (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:49.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:47.835137+0000 mon.smithi133 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.134:0/2021190641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:49.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:49.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:50.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:47.940678+0000 mgr.smithi133.xdyttl (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:50.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 -- 172.21.15.134:0/4126349687 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 msgr2=0x7f226c0f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 --2- 172.21.15.134:0/4126349687 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c0f6bc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2268009fb0 tx=0x7f2268009b30).stop 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 -- 172.21.15.134:0/4126349687 shutdown_connections 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 --2- 172.21.15.134:0/4126349687 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c0f6bc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 -- 172.21.15.134:0/4126349687 >> 172.21.15.134:0/4126349687 conn(0x7f226c0f0120 msgr2=0x7f226c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 -- 172.21.15.134:0/4126349687 shutdown_connections 2023-11-16T01:09:50.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.425+0000 7f2272e8f700 1 -- 172.21.15.134:0/4126349687 wait complete. 2023-11-16T01:09:50.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2272e8f700 1 Processor -- start 2023-11-16T01:09:50.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2272e8f700 1 -- start start 2023-11-16T01:09:50.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2272e8f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:50.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2272e8f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c106a10 con 0x7f226c0f67a0 2023-11-16T01:09:50.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2271e8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:50.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2271e8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:43258/0 (socket says 172.21.15.134:43258) 2023-11-16T01:09:50.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2271e8d700 1 -- 172.21.15.134:0/2584499770 learned_addr learned my addr 172.21.15.134:0/2584499770 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:50.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2271e8d700 1 -- 172.21.15.134:0/2584499770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2268005040 con 0x7f226c0f67a0 2023-11-16T01:09:50.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f2271e8d700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2268009f80 tx=0x7f226800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:50.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.426+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2268009320 con 0x7f226c0f67a0 2023-11-16T01:09:50.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c106c10 con 0x7f226c0f67a0 2023-11-16T01:09:50.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2268013ec0 con 0x7f226c0f67a0 2023-11-16T01:09:50.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f226801ec80 con 0x7f226c0f67a0 2023-11-16T01:09:50.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c1070a0 con 0x7f226c0f67a0 2023-11-16T01:09:50.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f226801e420 con 0x7f226c0f67a0 2023-11-16T01:09:50.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c045ad0 con 0x7f226c0f67a0 2023-11-16T01:09:50.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.427+0000 7f22627fc700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 0x7f2258032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:50.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.428+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2268040f50 con 0x7f226c0f67a0 2023-11-16T01:09:50.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.428+0000 7f227168c700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 0x7f2258032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:50.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.428+0000 7f227168c700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 0x7f2258032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:50.437 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.432+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2268013380 con 0x7f226c0f67a0 2023-11-16T01:09:50.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.634+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f226c107f50 con 0x7f226c0f67a0 2023-11-16T01:09:50.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.635+0000 7f22627fc700 1 -- 172.21.15.134:0/2584499770 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2268004490 con 0x7f226c0f67a0 2023-11-16T01:09:50.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:50.636 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 msgr2=0x7f2258032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 0x7f2258032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).stop 2023-11-16T01:09:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 msgr2=0x7f226c106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2268009f80 tx=0x7f226800b040).stop 2023-11-16T01:09:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 shutdown_connections 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225802ff60 0x7f2258032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 --2- 172.21.15.134:0/2584499770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f226c0f67a0 0x7f226c106520 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 >> 172.21.15.134:0/2584499770 conn(0x7f226c0f0120 msgr2=0x7f226c0f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 shutdown_connections 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:50.637+0000 7f2272e8f700 1 -- 172.21.15.134:0/2584499770 wait complete. 2023-11-16T01:09:50.638 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:51.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:50.635674+0000 mon.smithi133 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.134:0/2584499770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:52.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:49.940941+0000 mgr.smithi133.xdyttl (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:52.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:52.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:53.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 -- 172.21.15.134:0/382535625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 msgr2=0x7fdd1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 --2- 172.21.15.134:0/382535625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c0f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdd0c009fb0 tx=0x7fdd0c009b30).stop 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 -- 172.21.15.134:0/382535625 shutdown_connections 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 --2- 172.21.15.134:0/382535625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c0f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 -- 172.21.15.134:0/382535625 >> 172.21.15.134:0/382535625 conn(0x7fdd1c0f1c70 msgr2=0x7fdd1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 -- 172.21.15.134:0/382535625 shutdown_connections 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.174+0000 7fdd23b23700 1 -- 172.21.15.134:0/382535625 wait complete. 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd23b23700 1 Processor -- start 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd23b23700 1 -- start start 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd23b23700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:53.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd23b23700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c069e20 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd218bf700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd218bf700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:43278/0 (socket says 172.21.15.134:43278) 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd218bf700 1 -- 172.21.15.134:0/861520620 learned_addr learned my addr 172.21.15.134:0/861520620 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.175+0000 7fdd218bf700 1 -- 172.21.15.134:0/861520620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c005040 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd218bf700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdd0c013510 tx=0x7fdd0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd0c009320 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c069fc0 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd0c013f00 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd0c014d90 con 0x7fdd1c0f6510 2023-11-16T01:09:53.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c06a450 con 0x7fdd1c0f6510 2023-11-16T01:09:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.176+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd0c0095f0 con 0x7fdd1c0f6510 2023-11-16T01:09:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.177+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c059100 con 0x7fdd1c0f6510 2023-11-16T01:09:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.177+0000 7fdd127fc700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 0x7fdd08032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.177+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd0c040ff0 con 0x7fdd1c0f6510 2023-11-16T01:09:53.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.177+0000 7fdd210be700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 0x7fdd08032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:53.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.177+0000 7fdd210be700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 0x7fdd08032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fdd180096a0 tx=0x7fdd18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.181+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd0c06d2b0 con 0x7fdd1c0f6510 2023-11-16T01:09:53.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.389+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd1c0fbf60 con 0x7fdd1c0f6510 2023-11-16T01:09:53.390 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.389+0000 7fdd127fc700 1 -- 172.21.15.134:0/861520620 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd0c014420 con 0x7fdd1c0f6510 2023-11-16T01:09:53.390 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:53.390 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:53.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 msgr2=0x7fdd08032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:53.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 0x7fdd08032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fdd180096a0 tx=0x7fdd18006b40).stop 2023-11-16T01:09:53.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 msgr2=0x7fdd1c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:53.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdd0c013510 tx=0x7fdd0c00b040).stop 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 shutdown_connections 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdd0802ffb0 0x7fdd08032470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 --2- 172.21.15.134:0/861520620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd1c0f6510 0x7fdd1c06b9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 >> 172.21.15.134:0/861520620 conn(0x7fdd1c0f1c70 msgr2=0x7fdd1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 shutdown_connections 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:53.391+0000 7fdd23b23700 1 -- 172.21.15.134:0/861520620 wait complete. 2023-11-16T01:09:53.393 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:51.941277+0000 mgr.smithi133.xdyttl (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:53.389896+0000 mon.smithi133 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.134:0/861520620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:54.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:54.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- 172.21.15.134:0/193866328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 msgr2=0x7f70200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 --2- 172.21.15.134:0/193866328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f70200f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7014009fb0 tx=0x7f7014009b30).stop 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- 172.21.15.134:0/193866328 shutdown_connections 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 --2- 172.21.15.134:0/193866328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f70200f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- 172.21.15.134:0/193866328 >> 172.21.15.134:0/193866328 conn(0x7f70200f1c70 msgr2=0x7f70200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- 172.21.15.134:0/193866328 shutdown_connections 2023-11-16T01:09:55.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- 172.21.15.134:0/193866328 wait complete. 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 Processor -- start 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.932+0000 7f7026db6700 1 -- start start 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7026db6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7026db6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020106c20 con 0x7f70200f6510 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7024b52700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7024b52700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:43306/0 (socket says 172.21.15.134:43306) 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7024b52700 1 -- 172.21.15.134:0/3549218131 learned_addr learned my addr 172.21.15.134:0/3549218131 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7024b52700 1 -- 172.21.15.134:0/3549218131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7014005040 con 0x7f70200f6510 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.933+0000 7f7024b52700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7014013510 tx=0x7f701400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7014009320 con 0x7f70200f6510 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7014013f00 con 0x7f70200f6510 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020106e20 con 0x7f70200f6510 2023-11-16T01:09:55.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7014014d90 con 0x7f70200f6510 2023-11-16T01:09:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70201072b0 con 0x7f70200f6510 2023-11-16T01:09:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70140095f0 con 0x7f70200f6510 2023-11-16T01:09:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.934+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020045ad0 con 0x7f70200f6510 2023-11-16T01:09:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.935+0000 7f701d7fa700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 0x7f7010032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.935+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70140410c0 con 0x7f70200f6510 2023-11-16T01:09:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.935+0000 7f701ffff700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 0x7f7010032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.935+0000 7f701ffff700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 0x7f7010032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f700c0096a0 tx=0x7f700c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:55.939+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7014014420 con 0x7f70200f6510 2023-11-16T01:09:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:53.941646+0000 mgr.smithi133.xdyttl (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:56.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.145+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70200fbf60 con 0x7f70200f6510 2023-11-16T01:09:56.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.146+0000 7f701d7fa700 1 -- 172.21.15.134:0/3549218131 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f701401d7f0 con 0x7f70200f6510 2023-11-16T01:09:56.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:56.148 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:56.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 msgr2=0x7f7010032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:56.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 0x7f7010032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f700c0096a0 tx=0x7f700c006b40).stop 2023-11-16T01:09:56.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 msgr2=0x7f7020106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:56.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7014013510 tx=0x7f701400b040).stop 2023-11-16T01:09:56.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 shutdown_connections 2023-11-16T01:09:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f701002ff60 0x7f7010032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 --2- 172.21.15.134:0/3549218131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70200f6510 0x7f7020106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 >> 172.21.15.134:0/3549218131 conn(0x7f70200f1c70 msgr2=0x7f70200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.147+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 shutdown_connections 2023-11-16T01:09:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:56.148+0000 7f7026db6700 1 -- 172.21.15.134:0/3549218131 wait complete. 2023-11-16T01:09:56.152 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:09:57.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:56.147556+0000 mon.smithi133 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.134:0/3549218131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:09:57.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:09:57.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:09:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:55.941899+0000 mgr.smithi133.xdyttl (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:09:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 -- 172.21.15.134:0/2079281868 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 msgr2=0x7fa9500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 --2- 172.21.15.134:0/2079281868 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa9500f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa944009fb0 tx=0x7fa944009b30).stop 2023-11-16T01:09:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 -- 172.21.15.134:0/2079281868 shutdown_connections 2023-11-16T01:09:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 --2- 172.21.15.134:0/2079281868 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa9500f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 -- 172.21.15.134:0/2079281868 >> 172.21.15.134:0/2079281868 conn(0x7fa9500f1c70 msgr2=0x7fa9500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 -- 172.21.15.134:0/2079281868 shutdown_connections 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.694+0000 7fa956dde700 1 -- 172.21.15.134:0/2079281868 wait complete. 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa956dde700 1 Processor -- start 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa956dde700 1 -- start start 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa956dde700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:58.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa956dde700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa950069e20 con 0x7fa9500f6510 2023-11-16T01:09:58.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa954b7a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:58.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa954b7a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:43314/0 (socket says 172.21.15.134:43314) 2023-11-16T01:09:58.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa954b7a700 1 -- 172.21.15.134:0/3328686050 learned_addr learned my addr 172.21.15.134:0/3328686050 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:09:58.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa954b7a700 1 -- 172.21.15.134:0/3328686050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944005040 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.695+0000 7fa954b7a700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa944013510 tx=0x7fa94400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa944009320 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa944013f00 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa950069fc0 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa944014d90 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95006a450 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.696+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa944014420 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.697+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa950059100 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.697+0000 7fa94d7fa700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 0x7fa940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.697+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9440410c0 con 0x7fa9500f6510 2023-11-16T01:09:58.699 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.697+0000 7fa94ffff700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 0x7fa940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:09:58.700 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.697+0000 7fa94ffff700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 0x7fa940032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa93c0096a0 tx=0x7fa93c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:09:58.703 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.702+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9440095f0 con 0x7fa9500f6510 2023-11-16T01:09:58.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.912+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9500fbf60 con 0x7fa9500f6510 2023-11-16T01:09:58.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.913+0000 7fa94d7fa700 1 -- 172.21.15.134:0/3328686050 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa94401db70 con 0x7fa9500f6510 2023-11-16T01:09:58.987 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:09:58.987 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 msgr2=0x7fa940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 0x7fa940032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa93c0096a0 tx=0x7fa93c006b40).stop 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 msgr2=0x7fa95006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa944013510 tx=0x7fa94400b040).stop 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 shutdown_connections 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa94002ff60 0x7fa940032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 --2- 172.21.15.134:0/3328686050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9500f6510 0x7fa95006b9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:09:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 >> 172.21.15.134:0/3328686050 conn(0x7fa9500f1c70 msgr2=0x7fa9500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:09:58.989 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 shutdown_connections 2023-11-16T01:09:58.989 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:09:58.914+0000 7fa956dde700 1 -- 172.21.15.134:0/3328686050 wait complete. 2023-11-16T01:09:58.989 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:57.942244+0000 mgr.smithi133.xdyttl (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:09:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:09:58.914181+0000 mon.smithi133 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.134:0/3328686050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:00.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:00.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:01.592 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- 172.21.15.134:0/4089042050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 msgr2=0x7f9cc00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/4089042050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc00f66c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9cb0009fb0 tx=0x7f9cb0009b30).stop 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- 172.21.15.134:0/4089042050 shutdown_connections 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/4089042050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc00f66c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- 172.21.15.134:0/4089042050 >> 172.21.15.134:0/4089042050 conn(0x7f9cc00f1aa0 msgr2=0x7f9cc00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- 172.21.15.134:0/4089042050 shutdown_connections 2023-11-16T01:10:01.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- 172.21.15.134:0/4089042050 wait complete. 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 Processor -- start 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.588+0000 7f9cc4e09700 1 -- start start 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cc4e09700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cc4e09700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01069a0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cbf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cbf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47338/0 (socket says 172.21.15.134:47338) 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cbf7fe700 1 -- 172.21.15.134:0/374589290 learned_addr learned my addr 172.21.15.134:0/374589290 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cbf7fe700 1 -- 172.21.15.134:0/374589290 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0005040 con 0x7f9cc00f62a0 2023-11-16T01:10:01.594 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cbf7fe700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9cb0009830 tx=0x7f9cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb0009320 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc0106ba0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cb00136c0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb001ee90 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.589+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc0107030 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.590+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cb001e420 con 0x7f9cc00f62a0 2023-11-16T01:10:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.590+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0045ad0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.590+0000 7f9ca7fff700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 0x7f9ca80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:01.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.590+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cb0040fe0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.591+0000 7f9cbeffd700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 0x7f9ca80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:01.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.591+0000 7f9cbeffd700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 0x7f9ca80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9cb40096a0 tx=0x7f9cb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:01.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.595+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cb001c790 con 0x7f9cc00f62a0 2023-11-16T01:10:01.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.799+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cc0107ee0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.799+0000 7f9ca7fff700 1 -- 172.21.15.134:0/374589290 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cb002c2d0 con 0x7f9cc00f62a0 2023-11-16T01:10:01.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:01.801 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 msgr2=0x7f9ca80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 0x7f9ca80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9cb40096a0 tx=0x7f9cb4006b40).stop 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 msgr2=0x7f9cc01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9cb0009830 tx=0x7f9cb000b040).stop 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 shutdown_connections 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ca802ff10 0x7f9ca80323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 --2- 172.21.15.134:0/374589290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cc00f62a0 0x7f9cc01064b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:01.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 >> 172.21.15.134:0/374589290 conn(0x7f9cc00f1aa0 msgr2=0x7f9cc00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:01.804 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 shutdown_connections 2023-11-16T01:10:01.804 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:01.801+0000 7f9cc4e09700 1 -- 172.21.15.134:0/374589290 wait complete. 2023-11-16T01:10:01.804 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:09:59.942521+0000 mgr.smithi133.xdyttl (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:03.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:01.800837+0000 mon.smithi133 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.134:0/374589290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:03.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:03.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:01.942869+0000 mgr.smithi133.xdyttl (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:04.366 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 -- 172.21.15.134:0/1220929056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 msgr2=0x7f33000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:04.366 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 --2- 172.21.15.134:0/1220929056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f33000f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f32f0009fb0 tx=0x7f32f0009b30).stop 2023-11-16T01:10:04.366 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 -- 172.21.15.134:0/1220929056 shutdown_connections 2023-11-16T01:10:04.366 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 --2- 172.21.15.134:0/1220929056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f33000f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 -- 172.21.15.134:0/1220929056 >> 172.21.15.134:0/1220929056 conn(0x7f33000f1c70 msgr2=0x7f33000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 -- 172.21.15.134:0/1220929056 shutdown_connections 2023-11-16T01:10:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 -- 172.21.15.134:0/1220929056 wait complete. 2023-11-16T01:10:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.361+0000 7f3306277700 1 Processor -- start 2023-11-16T01:10:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f3306277700 1 -- start start 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f3306277700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f3306277700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3300106c20 con 0x7f33000f6510 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f32fffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f32fffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47364/0 (socket says 172.21.15.134:47364) 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f32fffff700 1 -- 172.21.15.134:0/2630553915 learned_addr learned my addr 172.21.15.134:0/2630553915 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f32fffff700 1 -- 172.21.15.134:0/2630553915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f0005040 con 0x7f33000f6510 2023-11-16T01:10:04.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.362+0000 7f32fffff700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f32f0013510 tx=0x7f32f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32f0009320 con 0x7f33000f6510 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3300106e20 con 0x7f33000f6510 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32f0013f00 con 0x7f33000f6510 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32f0014d90 con 0x7f33000f6510 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33001072b0 con 0x7f33000f6510 2023-11-16T01:10:04.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32f00263a0 con 0x7f33000f6510 2023-11-16T01:10:04.370 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.363+0000 7f32fcff9700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 0x7f32ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:04.370 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.364+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32f0040fc0 con 0x7f33000f6510 2023-11-16T01:10:04.370 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.364+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3300045ad0 con 0x7f33000f6510 2023-11-16T01:10:04.370 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.364+0000 7f32ff7fe700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 0x7f32ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:04.370 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.364+0000 7f32ff7fe700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 0x7f32ec032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f32e80096a0 tx=0x7f32e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:04.371 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.368+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32f0012920 con 0x7f33000f6510 2023-11-16T01:10:04.577 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.575+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3300107fd0 con 0x7f33000f6510 2023-11-16T01:10:04.578 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.576+0000 7f32fcff9700 1 -- 172.21.15.134:0/2630553915 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f32f002b020 con 0x7f33000f6510 2023-11-16T01:10:04.578 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:04.579 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 msgr2=0x7f32ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 0x7f32ec032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f32e80096a0 tx=0x7f32e8006b40).stop 2023-11-16T01:10:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 msgr2=0x7f3300106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f32f0013510 tx=0x7f32f000b040).stop 2023-11-16T01:10:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 shutdown_connections 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f32ec02ff60 0x7f32ec032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 --2- 172.21.15.134:0/2630553915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33000f6510 0x7f3300106730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 >> 172.21.15.134:0/2630553915 conn(0x7f33000f1c70 msgr2=0x7f33000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 shutdown_connections 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:04.578+0000 7f3306277700 1 -- 172.21.15.134:0/2630553915 wait complete. 2023-11-16T01:10:04.581 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:05.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:04 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:04.577746+0000 mon.smithi133 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.134:0/2630553915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:05.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:05.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:03.943239+0000 mgr.smithi133.xdyttl (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:07.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 -- 172.21.15.134:0/394892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 msgr2=0x7f96540f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:07.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 --2- 172.21.15.134:0/394892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96540f6850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9644009fb0 tx=0x7f9644009b30).stop 2023-11-16T01:10:07.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 -- 172.21.15.134:0/394892 shutdown_connections 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 --2- 172.21.15.134:0/394892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96540f6850 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 -- 172.21.15.134:0/394892 >> 172.21.15.134:0/394892 conn(0x7f96540f1b90 msgr2=0x7f96540f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 -- 172.21.15.134:0/394892 shutdown_connections 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.187+0000 7f965b4b4700 1 -- 172.21.15.134:0/394892 wait complete. 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f965b4b4700 1 Processor -- start 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f965b4b4700 1 -- start start 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f965b4b4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f965b4b4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654106ec0 con 0x7f96540f6430 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f9659250700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f9659250700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47382/0 (socket says 172.21.15.134:47382) 2023-11-16T01:10:07.190 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f9659250700 1 -- 172.21.15.134:0/4216069417 learned_addr learned my addr 172.21.15.134:0/4216069417 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f9659250700 1 -- 172.21.15.134:0/4216069417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644005040 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.188+0000 7f9659250700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9644013510 tx=0x7f964400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.189+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9644009320 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.189+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96541070c0 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.189+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9644013f00 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.189+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9644014d90 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.189+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9654107550 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.190+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96440263a0 con 0x7f96540f6430 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.190+0000 7f9649ffb700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 0x7f9640032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:07.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.190+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9654045ad0 con 0x7f96540f6430 2023-11-16T01:10:07.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.190+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9644040ff0 con 0x7f96540f6430 2023-11-16T01:10:07.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.190+0000 7f9658a4f700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 0x7f9640032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:07.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.191+0000 7f9658a4f700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 0x7f9640032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:07.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.195+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f964406d2b0 con 0x7f96540f6430 2023-11-16T01:10:07.400 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.399+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f965405ea50 con 0x7f96540f6430 2023-11-16T01:10:07.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.399+0000 7f9649ffb700 1 -- 172.21.15.134:0/4216069417 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f96440126f0 con 0x7f96540f6430 2023-11-16T01:10:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 msgr2=0x7f9640032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 0x7f9640032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f96500096a0 tx=0x7f9650006b40).stop 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 msgr2=0x7f96541069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9644013510 tx=0x7f964400b040).stop 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 shutdown_connections 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f964002ff60 0x7f9640032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 --2- 172.21.15.134:0/4216069417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96540f6430 0x7f96541069d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 >> 172.21.15.134:0/4216069417 conn(0x7f96540f1b90 msgr2=0x7f96540f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 shutdown_connections 2023-11-16T01:10:07.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:07.401+0000 7f965b4b4700 1 -- 172.21.15.134:0/4216069417 wait complete. 2023-11-16T01:10:07.404 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:05.943645+0000 mgr.smithi133.xdyttl (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:07.400903+0000 mon.smithi133 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.134:0/4216069417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:08.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:08.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:07.943995+0000 mgr.smithi133.xdyttl (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:10.047 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.044+0000 7fa0995b3700 1 -- 172.21.15.134:0/2120153201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 msgr2=0x7fa0940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.044+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2120153201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0940f55c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa084009fb0 tx=0x7fa084009b30).stop 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.044+0000 7fa0995b3700 1 -- 172.21.15.134:0/2120153201 shutdown_connections 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2120153201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0940f55c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 -- 172.21.15.134:0/2120153201 >> 172.21.15.134:0/2120153201 conn(0x7fa0940f09a0 msgr2=0x7fa0940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 -- 172.21.15.134:0/2120153201 shutdown_connections 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 -- 172.21.15.134:0/2120153201 wait complete. 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 Processor -- start 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 -- start start 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.045+0000 7fa0995b3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa094060240 con 0x7fa0940f51a0 2023-11-16T01:10:10.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa093fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa093fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49970/0 (socket says 172.21.15.134:49970) 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa093fff700 1 -- 172.21.15.134:0/2695518876 learned_addr learned my addr 172.21.15.134:0/2695518876 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa093fff700 1 -- 172.21.15.134:0/2695518876 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa084005040 con 0x7fa0940f51a0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa093fff700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fa084013510 tx=0x7fa08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa084009320 con 0x7fa0940f51a0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa084013f00 con 0x7fa0940f51a0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa094060440 con 0x7fa0940f51a0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa084014dc0 con 0x7fa0940f51a0 2023-11-16T01:10:10.049 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.046+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa094109860 con 0x7fa0940f51a0 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.047+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa084014420 con 0x7fa0940f51a0 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.047+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa094045ad0 con 0x7fa0940f51a0 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.047+0000 7fa090ff9700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.047+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa084041030 con 0x7fa0940f51a0 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.047+0000 7fa0937fe700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:10.050 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.048+0000 7fa0937fe700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 0x7fa07c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa0880096a0 tx=0x7fa088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:10.053 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.052+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa084026bf0 con 0x7fa0940f51a0 2023-11-16T01:10:10.256 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.255+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa09410a080 con 0x7fa0940f51a0 2023-11-16T01:10:10.257 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.255+0000 7fa090ff9700 1 -- 172.21.15.134:0/2695518876 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa08401cb10 con 0x7fa0940f51a0 2023-11-16T01:10:10.257 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:10.257 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 msgr2=0x7fa07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 0x7fa07c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa0880096a0 tx=0x7fa088006b40).stop 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 msgr2=0x7fa0941091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fa084013510 tx=0x7fa08400b040).stop 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 shutdown_connections 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 --2- 172.21.15.134:0/2695518876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0940f51a0 0x7fa0941091b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 >> 172.21.15.134:0/2695518876 conn(0x7fa0940f09a0 msgr2=0x7fa0940f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 shutdown_connections 2023-11-16T01:10:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:10.257+0000 7fa0995b3700 1 -- 172.21.15.134:0/2695518876 wait complete. 2023-11-16T01:10:10.260 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:11.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:10.257112+0000 mon.smithi133 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.134:0/2695518876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:11.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:11.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:09.944367+0000 mgr.smithi133.xdyttl (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:12.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 -- 172.21.15.134:0/434602606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 msgr2=0x7f37980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 --2- 172.21.15.134:0/434602606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f37980f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f378c009fb0 tx=0x7f378c009b30).stop 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 -- 172.21.15.134:0/434602606 shutdown_connections 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 --2- 172.21.15.134:0/434602606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f37980f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 -- 172.21.15.134:0/434602606 >> 172.21.15.134:0/434602606 conn(0x7f37980f0b70 msgr2=0x7f37980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 -- 172.21.15.134:0/434602606 shutdown_connections 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.896+0000 7f379ceb0700 1 -- 172.21.15.134:0/434602606 wait complete. 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379ceb0700 1 Processor -- start 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379ceb0700 1 -- start start 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379ceb0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379ceb0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3798105b20 con 0x7f37980f5410 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:12.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49986/0 (socket says 172.21.15.134:49986) 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.897+0000 7f379659c700 1 -- 172.21.15.134:0/1336208219 learned_addr learned my addr 172.21.15.134:0/1336208219 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f379659c700 1 -- 172.21.15.134:0/1336208219 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c005040 con 0x7f37980f5410 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f379659c700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f378c013510 tx=0x7f378c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c009320 con 0x7f37980f5410 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3798105d20 con 0x7f37980f5410 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f378c013f00 con 0x7f37980f5410 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378c014d90 con 0x7f37980f5410 2023-11-16T01:10:12.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.898+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37981061b0 con 0x7f37980f5410 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.899+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f378c014420 con 0x7f37980f5410 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.899+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798045ad0 con 0x7f37980f5410 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.899+0000 7f3786ffd700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 0x7f3780032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.899+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f378c042e60 con 0x7f37980f5410 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.899+0000 7f3795d9b700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 0x7f3780032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:12.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.900+0000 7f3795d9b700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 0x7f3780032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37880096a0 tx=0x7f3788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:12.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:12.904+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f378c0095f0 con 0x7f37980f5410 2023-11-16T01:10:13.112 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.110+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37980fae60 con 0x7f37980f5410 2023-11-16T01:10:13.112 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.111+0000 7f3786ffd700 1 -- 172.21.15.134:0/1336208219 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f378c01fb70 con 0x7f37980f5410 2023-11-16T01:10:13.112 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:13.112 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 msgr2=0x7f3780032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 0x7f3780032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37880096a0 tx=0x7f3788006b40).stop 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 msgr2=0x7f3798105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f378c013510 tx=0x7f378c00b040).stop 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 shutdown_connections 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f378002ff60 0x7f3780032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:13.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 --2- 172.21.15.134:0/1336208219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37980f5410 0x7f3798105630 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:13.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 >> 172.21.15.134:0/1336208219 conn(0x7f37980f0b70 msgr2=0x7f37980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:13.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 shutdown_connections 2023-11-16T01:10:13.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:13.112+0000 7f379ceb0700 1 -- 172.21.15.134:0/1336208219 wait complete. 2023-11-16T01:10:13.115 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:11.944736+0000 mgr.smithi133.xdyttl (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:13.112456+0000 mon.smithi133 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.134:0/1336208219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:14.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:14.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:15.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 -- 172.21.15.134:0/4254026637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 msgr2=0x7fc5b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 --2- 172.21.15.134:0/4254026637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fc5ac009fb0 tx=0x7fc5ac009b30).stop 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 -- 172.21.15.134:0/4254026637 shutdown_connections 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 --2- 172.21.15.134:0/4254026637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 -- 172.21.15.134:0/4254026637 >> 172.21.15.134:0/4254026637 conn(0x7fc5b80f0b70 msgr2=0x7fc5b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 -- 172.21.15.134:0/4254026637 shutdown_connections 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.742+0000 7fc5bf291700 1 -- 172.21.15.134:0/4254026637 wait complete. 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bf291700 1 Processor -- start 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bf291700 1 -- start start 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bf291700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bf291700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80fb820 con 0x7fc5b80f5410 2023-11-16T01:10:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bd02d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bd02d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:50004/0 (socket says 172.21.15.134:50004) 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bd02d700 1 -- 172.21.15.134:0/831938179 learned_addr learned my addr 172.21.15.134:0/831938179 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.743+0000 7fc5bd02d700 1 -- 172.21.15.134:0/831938179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac005040 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5bd02d700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac009320 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ac013f00 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80fb9c0 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac014d90 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.744+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80fbe50 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5ac0095f0 con 0x7fc5b80f5410 2023-11-16T01:10:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8059100 con 0x7fc5b80f5410 2023-11-16T01:10:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5a9ffb700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5ac041090 con 0x7fc5b80f5410 2023-11-16T01:10:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5bc82c700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.745+0000 7fc5bc82c700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc5b40096a0 tx=0x7fc5b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:15.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.749+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5ac014420 con 0x7fc5b80f5410 2023-11-16T01:10:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.955+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5b80fae60 con 0x7fc5b80f5410 2023-11-16T01:10:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.956+0000 7fc5a9ffb700 1 -- 172.21.15.134:0/831938179 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc5ac01d7f0 con 0x7fc5b80f5410 2023-11-16T01:10:15.957 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:15.957 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 msgr2=0x7fc5a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc5b40096a0 tx=0x7fc5b4006b40).stop 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 msgr2=0x7fc5b80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).stop 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 shutdown_connections 2023-11-16T01:10:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:15.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 --2- 172.21.15.134:0/831938179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5b80f5410 0x7fc5b80fd3a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:15.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.957+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 >> 172.21.15.134:0/831938179 conn(0x7fc5b80f0b70 msgr2=0x7fc5b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:15.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.958+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 shutdown_connections 2023-11-16T01:10:15.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:15.958+0000 7fc5bf291700 1 -- 172.21.15.134:0/831938179 wait complete. 2023-11-16T01:10:15.960 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:16.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:13.945018+0000 mgr.smithi133.xdyttl (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:17.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:16 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:15.957171+0000 mon.smithi133 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.134:0/831938179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:17.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:17.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:15.945406+0000 mgr.smithi133.xdyttl (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 -- 172.21.15.134:0/1251640711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 msgr2=0x7f80e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 --2- 172.21.15.134:0/1251640711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e00f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f80d0009fb0 tx=0x7f80d0009b30).stop 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 -- 172.21.15.134:0/1251640711 shutdown_connections 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 --2- 172.21.15.134:0/1251640711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e00f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 -- 172.21.15.134:0/1251640711 >> 172.21.15.134:0/1251640711 conn(0x7f80e00f0b70 msgr2=0x7f80e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 -- 172.21.15.134:0/1251640711 shutdown_connections 2023-11-16T01:10:18.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.559+0000 7f80e7739700 1 -- 172.21.15.134:0/1251640711 wait complete. 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e7739700 1 Processor -- start 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e7739700 1 -- start start 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e7739700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e7739700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e0060220 con 0x7f80e00f5410 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e54d5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:18.563 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e54d5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:50022/0 (socket says 172.21.15.134:50022) 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e54d5700 1 -- 172.21.15.134:0/802075782 learned_addr learned my addr 172.21.15.134:0/802075782 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e54d5700 1 -- 172.21.15.134:0/802075782 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d0005040 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e54d5700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f80d0013510 tx=0x7f80d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80d0009320 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.560+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e0060420 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.561+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80d0013f00 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.561+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80d0014d90 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.561+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e0109b00 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.561+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80d00263a0 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.562+0000 7f80d5ffb700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 0x7f80cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.562+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80d0041390 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.562+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e0059100 con 0x7f80e00f5410 2023-11-16T01:10:18.564 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.562+0000 7f80e4cd4700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 0x7f80cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:18.565 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.562+0000 7f80e4cd4700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 0x7f80cc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f80dc0096a0 tx=0x7f80dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:18.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.566+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80d0012330 con 0x7f80e00f5410 2023-11-16T01:10:18.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.770+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80e010a1d0 con 0x7f80e00f5410 2023-11-16T01:10:18.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.771+0000 7f80d5ffb700 1 -- 172.21.15.134:0/802075782 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80d000f340 con 0x7f80e00f5410 2023-11-16T01:10:18.773 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:18.773 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 msgr2=0x7f80cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 0x7f80cc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f80dc0096a0 tx=0x7f80dc006b40).stop 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 msgr2=0x7f80e0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f80d0013510 tx=0x7f80d000b040).stop 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 shutdown_connections 2023-11-16T01:10:18.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80cc02ff60 0x7f80cc032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:18.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 --2- 172.21.15.134:0/802075782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80e00f5410 0x7f80e0109450 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:18.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 >> 172.21.15.134:0/802075782 conn(0x7f80e00f0b70 msgr2=0x7f80e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:18.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 shutdown_connections 2023-11-16T01:10:18.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:18.773+0000 7f80e7739700 1 -- 172.21.15.134:0/802075782 wait complete. 2023-11-16T01:10:18.775 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:17.945793+0000 mgr.smithi133.xdyttl (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:18.772654+0000 mon.smithi133 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.134:0/802075782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:20.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:20.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- 172.21.15.134:0/1047481637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 msgr2=0x7fe1040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/1047481637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe1040f6930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0f4009fb0 tx=0x7fe0f4009b30).stop 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- 172.21.15.134:0/1047481637 shutdown_connections 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/1047481637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe1040f6930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- 172.21.15.134:0/1047481637 >> 172.21.15.134:0/1047481637 conn(0x7fe1040f1c70 msgr2=0x7fe1040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- 172.21.15.134:0/1047481637 shutdown_connections 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- 172.21.15.134:0/1047481637 wait complete. 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 Processor -- start 2023-11-16T01:10:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.341+0000 7fe10bdd7700 1 -- start start 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe10bdd7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe10bdd7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe104069e20 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe109b73700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe109b73700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40264/0 (socket says 172.21.15.134:40264) 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe109b73700 1 -- 172.21.15.134:0/612320407 learned_addr learned my addr 172.21.15.134:0/612320407 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe109b73700 1 -- 172.21.15.134:0/612320407 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4005040 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe109b73700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe0f4013510 tx=0x7fe0f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f4009320 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f4013f00 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe104069fc0 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f4014d90 con 0x7fe1040f6510 2023-11-16T01:10:21.345 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.342+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10406a450 con 0x7fe1040f6510 2023-11-16T01:10:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.343+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0f4014420 con 0x7fe1040f6510 2023-11-16T01:10:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.343+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe104059100 con 0x7fe1040f6510 2023-11-16T01:10:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.343+0000 7fe0fa7fc700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 0x7fe0f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.343+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0f4041020 con 0x7fe1040f6510 2023-11-16T01:10:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.344+0000 7fe109372700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 0x7fe0f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:21.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.344+0000 7fe109372700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 0x7fe0f0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe1000096a0 tx=0x7fe100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:21.349 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.349+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0f40095f0 con 0x7fe1040f6510 2023-11-16T01:10:21.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.551+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1040fbf60 con 0x7fe1040f6510 2023-11-16T01:10:21.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.552+0000 7fe0fa7fc700 1 -- 172.21.15.134:0/612320407 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe0f401db70 con 0x7fe1040f6510 2023-11-16T01:10:21.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:21.553 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 msgr2=0x7fe0f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 0x7fe0f0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe1000096a0 tx=0x7fe100006b40).stop 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 msgr2=0x7fe10406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe0f4013510 tx=0x7fe0f400b040).stop 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 shutdown_connections 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe0f002ff60 0x7fe0f0032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 --2- 172.21.15.134:0/612320407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1040f6510 0x7fe10406b9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 >> 172.21.15.134:0/612320407 conn(0x7fe1040f1c70 msgr2=0x7fe1040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:21.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 shutdown_connections 2023-11-16T01:10:21.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:21.553+0000 7fe10bdd7700 1 -- 172.21.15.134:0/612320407 wait complete. 2023-11-16T01:10:21.556 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:22.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:19.946180+0000 mgr.smithi133.xdyttl (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:22.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:21.553410+0000 mon.smithi133 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.134:0/612320407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:22.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:22.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:21.946494+0000 mgr.smithi133.xdyttl (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:24.129 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 -- 172.21.15.134:0/808898659 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 msgr2=0x7fb0380f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 --2- 172.21.15.134:0/808898659 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0380f4c80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fb020009fb0 tx=0x7fb020009b30).stop 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 -- 172.21.15.134:0/808898659 shutdown_connections 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 --2- 172.21.15.134:0/808898659 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0380f4c80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 -- 172.21.15.134:0/808898659 >> 172.21.15.134:0/808898659 conn(0x7fb0380f02e0 msgr2=0x7fb0380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 -- 172.21.15.134:0/808898659 shutdown_connections 2023-11-16T01:10:24.130 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.129+0000 7fb03d867700 1 -- 172.21.15.134:0/808898659 wait complete. 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb03d867700 1 Processor -- start 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb03d867700 1 -- start start 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb03d867700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb03d867700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038106dd0 con 0x7fb0380f4860 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb036ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb036ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40274/0 (socket says 172.21.15.134:40274) 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb036ffd700 1 -- 172.21.15.134:0/1122587377 learned_addr learned my addr 172.21.15.134:0/1122587377 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:24.132 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb036ffd700 1 -- 172.21.15.134:0/1122587377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020005040 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.130+0000 7fb036ffd700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb020013510 tx=0x7fb02000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb020009320 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb038106fd0 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb020013f00 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb020014dc0 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb038107460 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.131+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb02001d3d0 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.132+0000 7fb02f7fe700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 0x7fb024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.132+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb020041040 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.132+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb038059100 con 0x7fb0380f4860 2023-11-16T01:10:24.133 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.132+0000 7fb0367fc700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 0x7fb024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:24.134 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.132+0000 7fb0367fc700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 0x7fb024032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb0280096a0 tx=0x7fb028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:24.137 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.137+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb020012920 con 0x7fb0380f4860 2023-11-16T01:10:24.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.343+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0380fc2d0 con 0x7fb0380f4860 2023-11-16T01:10:24.344 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.344+0000 7fb02f7fe700 1 -- 172.21.15.134:0/1122587377 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb02002b020 con 0x7fb0380f4860 2023-11-16T01:10:24.345 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:24.345 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:24.346 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 msgr2=0x7fb024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:24.346 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 0x7fb024032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb0280096a0 tx=0x7fb028006b40).stop 2023-11-16T01:10:24.346 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 msgr2=0x7fb0381068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb020013510 tx=0x7fb02000b040).stop 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 shutdown_connections 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb02402ff60 0x7fb024032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 --2- 172.21.15.134:0/1122587377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb0380f4860 0x7fb0381068e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 >> 172.21.15.134:0/1122587377 conn(0x7fb0380f02e0 msgr2=0x7fb0380f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 shutdown_connections 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:24.346+0000 7fb03d867700 1 -- 172.21.15.134:0/1122587377 wait complete. 2023-11-16T01:10:24.347 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:25.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:24 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:24.344667+0000 mon.smithi133 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.134:0/1122587377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:25.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:25.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:23.946732+0000 mgr.smithi133.xdyttl (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:26.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 -- 172.21.15.134:0/1463114317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 msgr2=0x7fa9280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 --2- 172.21.15.134:0/1463114317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa9280f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa910009fb0 tx=0x7fa910009b30).stop 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 -- 172.21.15.134:0/1463114317 shutdown_connections 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 --2- 172.21.15.134:0/1463114317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa9280f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 -- 172.21.15.134:0/1463114317 >> 172.21.15.134:0/1463114317 conn(0x7fa9280f0b70 msgr2=0x7fa9280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 -- 172.21.15.134:0/1463114317 shutdown_connections 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.958+0000 7fa92cd75700 1 -- 172.21.15.134:0/1463114317 wait complete. 2023-11-16T01:10:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92cd75700 1 Processor -- start 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92cd75700 1 -- start start 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92cd75700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92cd75700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928106c00 con 0x7fa9280f5410 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40298/0 (socket says 172.21.15.134:40298) 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92659c700 1 -- 172.21.15.134:0/797401597 learned_addr learned my addr 172.21.15.134:0/797401597 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.959+0000 7fa92659c700 1 -- 172.21.15.134:0/797401597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910005040 con 0x7fa9280f5410 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa92659c700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa910013510 tx=0x7fa91000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910009320 con 0x7fa9280f5410 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa928106e00 con 0x7fa9280f5410 2023-11-16T01:10:26.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa910013f00 con 0x7fa9280f5410 2023-11-16T01:10:26.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa910014d90 con 0x7fa9280f5410 2023-11-16T01:10:26.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa928107290 con 0x7fa9280f5410 2023-11-16T01:10:26.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.960+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa910014420 con 0x7fa9280f5410 2023-11-16T01:10:26.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.961+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928045ad0 con 0x7fa9280f5410 2023-11-16T01:10:26.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.961+0000 7fa91effd700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 0x7fa914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:26.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.961+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9100410c0 con 0x7fa9280f5410 2023-11-16T01:10:26.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.961+0000 7fa925d9b700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 0x7fa914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:26.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.961+0000 7fa925d9b700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 0x7fa914032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa9180096a0 tx=0x7fa918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:26.965 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:26.965+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9100095f0 con 0x7fa9280f5410 2023-11-16T01:10:27.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.173+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9280fae60 con 0x7fa9280f5410 2023-11-16T01:10:27.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.174+0000 7fa91effd700 1 -- 172.21.15.134:0/797401597 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa91001db70 con 0x7fa9280f5410 2023-11-16T01:10:27.174 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:27.175 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:27.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.175+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 msgr2=0x7fa914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:27.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.175+0000 7fa92cd75700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 0x7fa914032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa9180096a0 tx=0x7fa918006b40).stop 2023-11-16T01:10:27.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.175+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 msgr2=0x7fa928106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:27.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.175+0000 7fa92cd75700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa910013510 tx=0x7fa91000b040).stop 2023-11-16T01:10:27.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 shutdown_connections 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa91402ff60 0x7fa914032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 --2- 172.21.15.134:0/797401597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9280f5410 0x7fa928106710 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 >> 172.21.15.134:0/797401597 conn(0x7fa9280f0b70 msgr2=0x7fa92805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 shutdown_connections 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:27.176+0000 7fa92cd75700 1 -- 172.21.15.134:0/797401597 wait complete. 2023-11-16T01:10:27.177 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:25.947028+0000 mgr.smithi133.xdyttl (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:27.174522+0000 mon.smithi133 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.134:0/797401597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:28.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:28.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:29.794 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 -- 172.21.15.134:0/3636179971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 msgr2=0x7efc080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3636179971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc080f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7efbf8009fb0 tx=0x7efbf8009b30).stop 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 -- 172.21.15.134:0/3636179971 shutdown_connections 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3636179971 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc080f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 -- 172.21.15.134:0/3636179971 >> 172.21.15.134:0/3636179971 conn(0x7efc080f0b70 msgr2=0x7efc080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 -- 172.21.15.134:0/3636179971 shutdown_connections 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 -- 172.21.15.134:0/3636179971 wait complete. 2023-11-16T01:10:29.795 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.793+0000 7efc0ea20700 1 Processor -- start 2023-11-16T01:10:29.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc0ea20700 1 -- start start 2023-11-16T01:10:29.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc0ea20700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:29.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc0ea20700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08106c10 con 0x7efc080f5410 2023-11-16T01:10:29.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc07fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:29.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc07fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:39358/0 (socket says 172.21.15.134:39358) 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc07fff700 1 -- 172.21.15.134:0/3273109296 learned_addr learned my addr 172.21.15.134:0/3273109296 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc07fff700 1 -- 172.21.15.134:0/3273109296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8005040 con 0x7efc080f5410 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.794+0000 7efc07fff700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efbf8013510 tx=0x7efbf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf8009320 con 0x7efc080f5410 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc08106e10 con 0x7efc080f5410 2023-11-16T01:10:29.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf8013f00 con 0x7efc080f5410 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf8014d90 con 0x7efc080f5410 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc081072a0 con 0x7efc080f5410 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efbf80263a0 con 0x7efc080f5410 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc04ff9700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 0x7efbf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.795+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08045ad0 con 0x7efc080f5410 2023-11-16T01:10:29.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.796+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efbf80410e0 con 0x7efc080f5410 2023-11-16T01:10:29.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.796+0000 7efc077fe700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 0x7efbf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:29.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.796+0000 7efc077fe700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 0x7efbf0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efbfc0096a0 tx=0x7efbfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:29.803 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:29.800+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbf8012920 con 0x7efc080f5410 2023-11-16T01:10:30.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.005+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc08108150 con 0x7efc080f5410 2023-11-16T01:10:30.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.005+0000 7efc04ff9700 1 -- 172.21.15.134:0/3273109296 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efbf800f340 con 0x7efc080f5410 2023-11-16T01:10:30.007 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:30.007 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:30.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 msgr2=0x7efbf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:30.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 0x7efbf0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efbfc0096a0 tx=0x7efbfc006b40).stop 2023-11-16T01:10:30.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 msgr2=0x7efc08106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:30.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efbf8013510 tx=0x7efbf800b040).stop 2023-11-16T01:10:30.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 shutdown_connections 2023-11-16T01:10:30.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efbf002ff60 0x7efbf0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:30.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 --2- 172.21.15.134:0/3273109296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc080f5410 0x7efc08106720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:30.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 >> 172.21.15.134:0/3273109296 conn(0x7efc080f0b70 msgr2=0x7efc0805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:30.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 shutdown_connections 2023-11-16T01:10:30.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:30.007+0000 7efc0ea20700 1 -- 172.21.15.134:0/3273109296 wait complete. 2023-11-16T01:10:30.011 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:30.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:27.947425+0000 mgr.smithi133.xdyttl (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:30.005930+0000 mon.smithi133 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.134:0/3273109296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:30.071884+0000 mon.smithi133 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:10:31.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:30.073555+0000 mon.smithi133 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:10:31.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:31.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:32.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:29.947746+0000 mgr.smithi133.xdyttl (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:32.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- 172.21.15.134:0/1910564782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 msgr2=0x7ff5d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 --2- 172.21.15.134:0/1910564782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d80f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff5cc009fb0 tx=0x7ff5cc009b30).stop 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- 172.21.15.134:0/1910564782 shutdown_connections 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 --2- 172.21.15.134:0/1910564782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d80f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- 172.21.15.134:0/1910564782 >> 172.21.15.134:0/1910564782 conn(0x7ff5d80f09a0 msgr2=0x7ff5d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- 172.21.15.134:0/1910564782 shutdown_connections 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- 172.21.15.134:0/1910564782 wait complete. 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 Processor -- start 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- start start 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d759e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8060220 con 0x7ff5d80f51a0 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:32.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:39378/0 (socket says 172.21.15.134:39378) 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.632+0000 7ff5d659c700 1 -- 172.21.15.134:0/4031139322 learned_addr learned my addr 172.21.15.134:0/4031139322 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5d659c700 1 -- 172.21.15.134:0/4031139322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc005040 con 0x7ff5d80f51a0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5d659c700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff5cc009830 tx=0x7ff5cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5cc009320 con 0x7ff5d80f51a0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5cc0136c0 con 0x7ff5d80f51a0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d8060420 con 0x7ff5d80f51a0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5cc01ee90 con 0x7ff5d80f51a0 2023-11-16T01:10:32.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.633+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d8109880 con 0x7ff5d80f51a0 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.634+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5cc01e420 con 0x7ff5d80f51a0 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.634+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d8059100 con 0x7ff5d80f51a0 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.634+0000 7ff5c6ffd700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.634+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5cc040ee0 con 0x7ff5d80f51a0 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.634+0000 7ff5d5d9b700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:32.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.635+0000 7ff5d5d9b700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 0x7ff5c00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff5c80096a0 tx=0x7ff5c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:32.639 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.639+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5cc0095f0 con 0x7ff5d80f51a0 2023-11-16T01:10:32.844 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.843+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5d8045ad0 con 0x7ff5d80f51a0 2023-11-16T01:10:32.844 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.844+0000 7ff5c6ffd700 1 -- 172.21.15.134:0/4031139322 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff5cc02c2d0 con 0x7ff5d80f51a0 2023-11-16T01:10:32.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:32.845 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:32.846 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 msgr2=0x7ff5c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:32.846 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 0x7ff5c00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff5c80096a0 tx=0x7ff5c8006b40).stop 2023-11-16T01:10:32.846 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 msgr2=0x7ff5d81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff5cc009830 tx=0x7ff5cc00b040).stop 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 shutdown_connections 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 --2- 172.21.15.134:0/4031139322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5d80f51a0 0x7ff5d81091d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 >> 172.21.15.134:0/4031139322 conn(0x7ff5d80f09a0 msgr2=0x7ff5d80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 shutdown_connections 2023-11-16T01:10:32.847 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:32.846+0000 7ff5d759e700 1 -- 172.21.15.134:0/4031139322 wait complete. 2023-11-16T01:10:32.848 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:31.948175+0000 mgr.smithi133.xdyttl (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:32.844489+0000 mon.smithi133 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.134:0/4031139322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:34.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:34.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:35.500 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- 172.21.15.134:0/1408231625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 msgr2=0x7f37e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:35.500 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 --2- 172.21.15.134:0/1408231625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e80f55c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f37dc009fb0 tx=0x7f37dc009b30).stop 2023-11-16T01:10:35.500 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- 172.21.15.134:0/1408231625 shutdown_connections 2023-11-16T01:10:35.500 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 --2- 172.21.15.134:0/1408231625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e80f55c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:35.500 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- 172.21.15.134:0/1408231625 >> 172.21.15.134:0/1408231625 conn(0x7f37e80f09a0 msgr2=0x7f37e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- 172.21.15.134:0/1408231625 shutdown_connections 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- 172.21.15.134:0/1408231625 wait complete. 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 Processor -- start 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- start start 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.499+0000 7f37edef8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e8060240 con 0x7f37e80f51a0 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37ecef6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37ecef6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:39392/0 (socket says 172.21.15.134:39392) 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37ecef6700 1 -- 172.21.15.134:0/3101230434 learned_addr learned my addr 172.21.15.134:0/3101230434 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37ecef6700 1 -- 172.21.15.134:0/3101230434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc005040 con 0x7f37e80f51a0 2023-11-16T01:10:35.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37ecef6700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f37dc013510 tx=0x7f37dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:35.502 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc009320 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e8060440 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc013f00 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc014dc0 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.500+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e8109860 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.501+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37dc014420 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.501+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e8045ad0 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.501+0000 7f37e57fa700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 0x7f37d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.501+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37dc041020 con 0x7f37e80f51a0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.501+0000 7f37e7fff700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 0x7f37d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:35.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.502+0000 7f37e7fff700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 0x7f37d8032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37d40096a0 tx=0x7f37d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:35.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.506+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37dc0095f0 con 0x7f37e80f51a0 2023-11-16T01:10:35.710 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.710+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37e810a070 con 0x7f37e80f51a0 2023-11-16T01:10:35.711 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.710+0000 7f37e57fa700 1 -- 172.21.15.134:0/3101230434 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37dc01cb10 con 0x7f37e80f51a0 2023-11-16T01:10:35.711 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:35.711 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 msgr2=0x7f37d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 0x7f37d8032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37d40096a0 tx=0x7f37d4006b40).stop 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 msgr2=0x7f37e81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f37dc013510 tx=0x7f37dc00b040).stop 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 shutdown_connections 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f37d802ff60 0x7f37d8032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 --2- 172.21.15.134:0/3101230434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37e80f51a0 0x7f37e81091b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 >> 172.21.15.134:0/3101230434 conn(0x7f37e80f09a0 msgr2=0x7f37e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 shutdown_connections 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:35.712+0000 7f37edef8700 1 -- 172.21.15.134:0/3101230434 wait complete. 2023-11-16T01:10:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:33.948508+0000 mgr.smithi133.xdyttl (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:35.710849+0000 mon.smithi133 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.134:0/3101230434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:37.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:37.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:38.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:35.948735+0000 mgr.smithi133.xdyttl (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:38.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.298+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3178603086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 msgr2=0x7f1d840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.298+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3178603086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d840f90f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f1d80009fb0 tx=0x7f1d80009b30).stop 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.298+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3178603086 shutdown_connections 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.298+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3178603086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d840f90f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3178603086 >> 172.21.15.134:0/3178603086 conn(0x7f1d840f0b70 msgr2=0x7f1d840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3178603086 shutdown_connections 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3178603086 wait complete. 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 Processor -- start 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 -- start start 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:38.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8c5cd700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d84106bd0 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8a369700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8a369700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:39404/0 (socket says 172.21.15.134:39404) 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.299+0000 7f1d8a369700 1 -- 172.21.15.134:0/3421106979 learned_addr learned my addr 172.21.15.134:0/3421106979 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d8a369700 1 -- 172.21.15.134:0/3421106979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d80005040 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d8a369700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1d80013510 tx=0x7f1d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d80009320 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d84106dd0 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d80013f00 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d80014dc0 con 0x7f1d84060280 2023-11-16T01:10:38.303 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.300+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d84107260 con 0x7f1d84060280 2023-11-16T01:10:38.304 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.301+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d8001d3d0 con 0x7f1d84060280 2023-11-16T01:10:38.304 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.301+0000 7f1d7affd700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 0x7f1d70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:38.304 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.301+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d80041050 con 0x7f1d84060280 2023-11-16T01:10:38.304 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.301+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d84059100 con 0x7f1d84060280 2023-11-16T01:10:38.305 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.301+0000 7f1d89b68700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 0x7f1d70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:38.305 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.302+0000 7f1d89b68700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 0x7f1d70032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1d740096a0 tx=0x7f1d74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:38.307 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.306+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d80012920 con 0x7f1d84060280 2023-11-16T01:10:38.513 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.511+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d84045ad0 con 0x7f1d84060280 2023-11-16T01:10:38.513 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.512+0000 7f1d7affd700 1 -- 172.21.15.134:0/3421106979 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d8002b020 con 0x7f1d84060280 2023-11-16T01:10:38.514 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:38.514 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.513+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 msgr2=0x7f1d70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.513+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 0x7f1d70032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1d740096a0 tx=0x7f1d74006b40).stop 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.513+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 msgr2=0x7f1d841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.513+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1d80013510 tx=0x7f1d8000b040).stop 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 shutdown_connections 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1d7002ff60 0x7f1d70032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:38.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 --2- 172.21.15.134:0/3421106979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d84060280 0x7f1d841066e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:38.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 >> 172.21.15.134:0/3421106979 conn(0x7f1d840f0b70 msgr2=0x7f1d840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:38.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 shutdown_connections 2023-11-16T01:10:38.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:38.514+0000 7f1d8c5cd700 1 -- 172.21.15.134:0/3421106979 wait complete. 2023-11-16T01:10:38.516 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:39.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:38.513581+0000 mon.smithi133 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.134:0/3421106979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:39.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:39.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:37.949111+0000 mgr.smithi133.xdyttl (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 -- 172.21.15.134:0/1543186490 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 msgr2=0x7f02e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 --2- 172.21.15.134:0/1543186490 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e80f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f02d8009fb0 tx=0x7f02d8009b30).stop 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 -- 172.21.15.134:0/1543186490 shutdown_connections 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 --2- 172.21.15.134:0/1543186490 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e80f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 -- 172.21.15.134:0/1543186490 >> 172.21.15.134:0/1543186490 conn(0x7f02e80f1c70 msgr2=0x7f02e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 -- 172.21.15.134:0/1543186490 shutdown_connections 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.098+0000 7f02efb28700 1 -- 172.21.15.134:0/1543186490 wait complete. 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02efb28700 1 Processor -- start 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02efb28700 1 -- start start 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02efb28700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02efb28700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8108e10 con 0x7f02e80f6510 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02ed8c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02ed8c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55008/0 (socket says 172.21.15.134:55008) 2023-11-16T01:10:41.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02ed8c4700 1 -- 172.21.15.134:0/3501130176 learned_addr learned my addr 172.21.15.134:0/3501130176 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.099+0000 7f02ed8c4700 1 -- 172.21.15.134:0/3501130176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8005040 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02ed8c4700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f02d8013510 tx=0x7f02d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02d8009320 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e8109010 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02d8013f00 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02d8014d90 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.100+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e81094a0 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02d80095f0 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8045ad0 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02de7fc700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 0x7f02d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02d80410e0 con 0x7f02e80f6510 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02ed0c3700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 0x7f02d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.101+0000 7f02ed0c3700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 0x7f02d4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:41.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.105+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02d8014420 con 0x7f02e80f6510 2023-11-16T01:10:41.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.309+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02e810a350 con 0x7f02e80f6510 2023-11-16T01:10:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.309+0000 7f02de7fc700 1 -- 172.21.15.134:0/3501130176 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f02d801d7f0 con 0x7f02e80f6510 2023-11-16T01:10:41.312 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:41.312 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:41.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 msgr2=0x7f02d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:41.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 0x7f02d4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).stop 2023-11-16T01:10:41.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 msgr2=0x7f02e8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:41.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f02d8013510 tx=0x7f02d800b040).stop 2023-11-16T01:10:41.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 shutdown_connections 2023-11-16T01:10:41.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f02d402ff60 0x7f02d4032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:41.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 --2- 172.21.15.134:0/3501130176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02e80f6510 0x7f02e8108920 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:41.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 >> 172.21.15.134:0/3501130176 conn(0x7f02e80f1c70 msgr2=0x7f02e80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:41.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 shutdown_connections 2023-11-16T01:10:41.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:41.311+0000 7f02efb28700 1 -- 172.21.15.134:0/3501130176 wait complete. 2023-11-16T01:10:41.315 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:42.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:39.949505+0000 mgr.smithi133.xdyttl (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:42.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:41.311041+0000 mon.smithi133 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.134:0/3501130176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:42.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:42.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:43.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 -- 172.21.15.134:0/773569201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 msgr2=0x7f69e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:43.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 --2- 172.21.15.134:0/773569201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e00f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f69d8009fb0 tx=0x7f69d8009b30).stop 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 -- 172.21.15.134:0/773569201 shutdown_connections 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 --2- 172.21.15.134:0/773569201 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e00f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 -- 172.21.15.134:0/773569201 >> 172.21.15.134:0/773569201 conn(0x7f69e00f0b70 msgr2=0x7f69e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 -- 172.21.15.134:0/773569201 shutdown_connections 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.896+0000 7f69e4e34700 1 -- 172.21.15.134:0/773569201 wait complete. 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69e4e34700 1 Processor -- start 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69e4e34700 1 -- start start 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69e4e34700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69e4e34700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0060220 con 0x7f69e00f5410 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69de59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69de59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55024/0 (socket says 172.21.15.134:55024) 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69de59c700 1 -- 172.21.15.134:0/59441473 learned_addr learned my addr 172.21.15.134:0/59441473 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69de59c700 1 -- 172.21.15.134:0/59441473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d8005040 con 0x7f69e00f5410 2023-11-16T01:10:43.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69de59c700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f69d8013510 tx=0x7f69d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d8009320 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0060420 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d8013f00 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.897+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d8014d90 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.898+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e0109b00 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.898+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69d80095f0 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.898+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e0059100 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.898+0000 7f69d6ffd700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 0x7f69cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.898+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69d80410a0 con 0x7f69e00f5410 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.899+0000 7f69ddd9b700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 0x7f69cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.899+0000 7f69ddd9b700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 0x7f69cc032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f69c80096a0 tx=0x7f69c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:43.903+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69d801d7f0 con 0x7f69e00f5410 2023-11-16T01:10:44.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:41.949884+0000 mgr.smithi133.xdyttl (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:44.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.109+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69e0045ad0 con 0x7f69e00f5410 2023-11-16T01:10:44.111 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.110+0000 7f69d6ffd700 1 -- 172.21.15.134:0/59441473 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69d8010230 con 0x7f69e00f5410 2023-11-16T01:10:44.112 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:44.112 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:44.113 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 msgr2=0x7f69cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 0x7f69cc032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f69c80096a0 tx=0x7f69c8006b40).stop 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 msgr2=0x7f69e0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f69d8013510 tx=0x7f69d800b040).stop 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 shutdown_connections 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69cc02ff60 0x7f69cc032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 --2- 172.21.15.134:0/59441473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69e00f5410 0x7f69e0109450 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 >> 172.21.15.134:0/59441473 conn(0x7f69e00f0b70 msgr2=0x7f69e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 shutdown_connections 2023-11-16T01:10:44.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:44.112+0000 7f69e4e34700 1 -- 172.21.15.134:0/59441473 wait complete. 2023-11-16T01:10:44.115 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:45.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:44 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:44.111432+0000 mon.smithi133 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.134:0/59441473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:45.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:45.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:46.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:43.950193+0000 mgr.smithi133.xdyttl (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 -- 172.21.15.134:0/1966793496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 msgr2=0x7f85380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 --2- 172.21.15.134:0/1966793496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f85380f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f852c009fb0 tx=0x7f852c009b30).stop 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 -- 172.21.15.134:0/1966793496 shutdown_connections 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 --2- 172.21.15.134:0/1966793496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f85380f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 -- 172.21.15.134:0/1966793496 >> 172.21.15.134:0/1966793496 conn(0x7f85380f0b70 msgr2=0x7f85380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 -- 172.21.15.134:0/1966793496 shutdown_connections 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 -- 172.21.15.134:0/1966793496 wait complete. 2023-11-16T01:10:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.666+0000 7f853e57e700 1 Processor -- start 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f853e57e700 1 -- start start 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f853e57e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f853e57e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538060240 con 0x7f85380f5410 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f8537fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f8537fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55052/0 (socket says 172.21.15.134:55052) 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f8537fff700 1 -- 172.21.15.134:0/1937236336 learned_addr learned my addr 172.21.15.134:0/1937236336 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.667+0000 7f8537fff700 1 -- 172.21.15.134:0/1937236336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c005040 con 0x7f85380f5410 2023-11-16T01:10:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f8537fff700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f852c013510 tx=0x7f852c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852c009320 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8538060440 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f852c013f00 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852c014d90 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.668+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8538109ae0 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f852c0095f0 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538045ad0 con 0x7f85380f5410 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f8534ff9700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:46.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f852c0410c0 con 0x7f85380f5410 2023-11-16T01:10:46.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f85377fe700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:46.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.669+0000 7f85377fe700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 0x7f8520032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f85280096a0 tx=0x7f8528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:46.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.673+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f852c01d7f0 con 0x7f85380f5410 2023-11-16T01:10:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.880+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85380fae60 con 0x7f85380f5410 2023-11-16T01:10:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.881+0000 7f8534ff9700 1 -- 172.21.15.134:0/1937236336 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f852c010230 con 0x7f85380f5410 2023-11-16T01:10:46.882 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:46.883 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 msgr2=0x7f8520032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 0x7f8520032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f85280096a0 tx=0x7f8528006b40).stop 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 msgr2=0x7f8538109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f852c013510 tx=0x7f852c00b040).stop 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 shutdown_connections 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 --2- 172.21.15.134:0/1937236336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85380f5410 0x7f8538109430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 >> 172.21.15.134:0/1937236336 conn(0x7f85380f0b70 msgr2=0x7f85380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 shutdown_connections 2023-11-16T01:10:46.885 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:46.882+0000 7f853e57e700 1 -- 172.21.15.134:0/1937236336 wait complete. 2023-11-16T01:10:46.885 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:45.950516+0000 mgr.smithi133.xdyttl (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:46.882403+0000 mon.smithi133 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.134:0/1937236336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:48.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:48.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 -- 172.21.15.134:0/324266454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 msgr2=0x7fe57c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 --2- 172.21.15.134:0/324266454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c0f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe56c009fb0 tx=0x7fe56c009b30).stop 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 -- 172.21.15.134:0/324266454 shutdown_connections 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 --2- 172.21.15.134:0/324266454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c0f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 -- 172.21.15.134:0/324266454 >> 172.21.15.134:0/324266454 conn(0x7fe57c0f0b70 msgr2=0x7fe57c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:49.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 -- 172.21.15.134:0/324266454 shutdown_connections 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.484+0000 7fe584097700 1 -- 172.21.15.134:0/324266454 wait complete. 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe584097700 1 Processor -- start 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe584097700 1 -- start start 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe584097700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe584097700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c106c10 con 0x7fe57c0f5410 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe581e33700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:49.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe581e33700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55026/0 (socket says 172.21.15.134:55026) 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe581e33700 1 -- 172.21.15.134:0/4202724647 learned_addr learned my addr 172.21.15.134:0/4202724647 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe581e33700 1 -- 172.21.15.134:0/4202724647 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c005040 con 0x7fe57c0f5410 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.485+0000 7fe581e33700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe56c013510 tx=0x7fe56c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c009320 con 0x7fe57c0f5410 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe56c013f00 con 0x7fe57c0f5410 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c106e10 con 0x7fe57c0f5410 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c014d90 con 0x7fe57c0f5410 2023-11-16T01:10:49.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c1072a0 con 0x7fe57c0f5410 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe56c0263a0 con 0x7fe57c0f5410 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.486+0000 7fe5727fc700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 0x7fe568032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.487+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe56c041080 con 0x7fe57c0f5410 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.487+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c045ad0 con 0x7fe57c0f5410 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.487+0000 7fe581632700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 0x7fe568032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:49.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.487+0000 7fe581632700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 0x7fe568032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe5780096a0 tx=0x7fe578006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:49.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.491+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe56c012920 con 0x7fe57c0f5410 2023-11-16T01:10:49.695 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.693+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe57c107ff0 con 0x7fe57c0f5410 2023-11-16T01:10:49.695 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.694+0000 7fe5727fc700 1 -- 172.21.15.134:0/4202724647 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe56c02b020 con 0x7fe57c0f5410 2023-11-16T01:10:49.695 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.695+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 msgr2=0x7fe568032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.695+0000 7fe584097700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 0x7fe568032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe5780096a0 tx=0x7fe578006b40).stop 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 msgr2=0x7fe57c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe56c013510 tx=0x7fe56c00b040).stop 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 shutdown_connections 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe56802ff60 0x7fe568032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 --2- 172.21.15.134:0/4202724647 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe57c0f5410 0x7fe57c106720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:49.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 >> 172.21.15.134:0/4202724647 conn(0x7fe57c0f0b70 msgr2=0x7fe57c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:49.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 shutdown_connections 2023-11-16T01:10:49.698 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:49.696+0000 7fe584097700 1 -- 172.21.15.134:0/4202724647 wait complete. 2023-11-16T01:10:49.698 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:50.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:47.950774+0000 mgr.smithi133.xdyttl (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:50.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:49.695664+0000 mon.smithi133 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.134:0/4202724647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:51.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:51.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:52.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:49.951109+0000 mgr.smithi133.xdyttl (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 -- 172.21.15.134:0/3080783603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 msgr2=0x7f8eb40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 --2- 172.21.15.134:0/3080783603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f90f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8ea8009fb0 tx=0x7f8ea8009b30).stop 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 -- 172.21.15.134:0/3080783603 shutdown_connections 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 --2- 172.21.15.134:0/3080783603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f90f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 -- 172.21.15.134:0/3080783603 >> 172.21.15.134:0/3080783603 conn(0x7f8eb40f0b70 msgr2=0x7f8eb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 -- 172.21.15.134:0/3080783603 shutdown_connections 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.307+0000 7f8ebab81700 1 -- 172.21.15.134:0/3080783603 wait complete. 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8ebab81700 1 Processor -- start 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8ebab81700 1 -- start start 2023-11-16T01:10:52.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8ebab81700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8ebab81700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40f4ff0 con 0x7f8eb4060280 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8eb891d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8eb891d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55046/0 (socket says 172.21.15.134:55046) 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8eb891d700 1 -- 172.21.15.134:0/4236730093 learned_addr learned my addr 172.21.15.134:0/4236730093 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.308+0000 7f8eb891d700 1 -- 172.21.15.134:0/4236730093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8005040 con 0x7f8eb4060280 2023-11-16T01:10:52.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8eb891d700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8ea8013510 tx=0x7f8ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea8009320 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb40f5190 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ea8013f00 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea8014dc0 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb40f5620 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.309+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ea80095f0 con 0x7f8eb4060280 2023-11-16T01:10:52.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.310+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb4059100 con 0x7f8eb4060280 2023-11-16T01:10:52.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.310+0000 7f8eb17fa700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:52.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.310+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ea8041100 con 0x7f8eb4060280 2023-11-16T01:10:52.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.310+0000 7f8eb3fff700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:52.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.310+0000 7f8eb3fff700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 0x7f8ea4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:52.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.315+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ea8014420 con 0x7f8eb4060280 2023-11-16T01:10:52.523 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.521+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8eb40fbf50 con 0x7f8eb4060280 2023-11-16T01:10:52.523 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.522+0000 7f8eb17fa700 1 -- 172.21.15.134:0/4236730093 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ea8010230 con 0x7f8eb4060280 2023-11-16T01:10:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:52.523 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 msgr2=0x7f8ea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 0x7f8ea4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).stop 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 msgr2=0x7f8eb40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8ea8013510 tx=0x7f8ea800b040).stop 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 shutdown_connections 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 --2- 172.21.15.134:0/4236730093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8eb4060280 0x7f8eb40f6b70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 >> 172.21.15.134:0/4236730093 conn(0x7f8eb40f0b70 msgr2=0x7f8eb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 shutdown_connections 2023-11-16T01:10:52.525 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:52.523+0000 7f8ebab81700 1 -- 172.21.15.134:0/4236730093 wait complete. 2023-11-16T01:10:52.526 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:53.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:52 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:52.523257+0000 mon.smithi133 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.134:0/4236730093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:53.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:53.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:51.951446+0000 mgr.smithi133.xdyttl (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:55.123 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- 172.21.15.134:0/807114779 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 msgr2=0x7fcaa00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:55.123 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/807114779 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa00f7d00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fca94009fb0 tx=0x7fca94009b30).stop 2023-11-16T01:10:55.123 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- 172.21.15.134:0/807114779 shutdown_connections 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/807114779 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa00f7d00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- 172.21.15.134:0/807114779 >> 172.21.15.134:0/807114779 conn(0x7fcaa00f0120 msgr2=0x7fcaa00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- 172.21.15.134:0/807114779 shutdown_connections 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- 172.21.15.134:0/807114779 wait complete. 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 Processor -- start 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.119+0000 7fcaa59e2700 1 -- start start 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa59e2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:55.124 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa59e2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa01069d0 con 0x7fcaa00f78e0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa49e0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa49e0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55062/0 (socket says 172.21.15.134:55062) 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa49e0700 1 -- 172.21.15.134:0/1839623458 learned_addr learned my addr 172.21.15.134:0/1839623458 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.120+0000 7fcaa49e0700 1 -- 172.21.15.134:0/1839623458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94005040 con 0x7fcaa00f78e0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fcaa49e0700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fca94013510 tx=0x7fca9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca94009320 con 0x7fcaa00f78e0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa0106bd0 con 0x7fcaa00f78e0 2023-11-16T01:10:55.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca94013f00 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca94014dc0 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa0107060 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.121+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca9401d3d0 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.122+0000 7fca9d7fa700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 0x7fca90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.122+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca94041090 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.122+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa0045ad0 con 0x7fcaa00f78e0 2023-11-16T01:10:55.126 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.122+0000 7fca9ffff700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 0x7fca90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:55.127 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.122+0000 7fca9ffff700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 0x7fca90032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fca8c0096a0 tx=0x7fca8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:55.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.127+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca94012920 con 0x7fcaa00f78e0 2023-11-16T01:10:55.338 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.337+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaa0107f10 con 0x7fcaa00f78e0 2023-11-16T01:10:55.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.337+0000 7fca9d7fa700 1 -- 172.21.15.134:0/1839623458 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca9402b020 con 0x7fcaa00f78e0 2023-11-16T01:10:55.339 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:55.340 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 msgr2=0x7fca90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 0x7fca90032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fca8c0096a0 tx=0x7fca8c006b40).stop 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 msgr2=0x7fcaa01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fca94013510 tx=0x7fca9400b040).stop 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 shutdown_connections 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fca9002ff60 0x7fca90032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 --2- 172.21.15.134:0/1839623458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcaa00f78e0 0x7fcaa01064e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 >> 172.21.15.134:0/1839623458 conn(0x7fcaa00f0120 msgr2=0x7fcaa00f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 shutdown_connections 2023-11-16T01:10:55.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:55.339+0000 7fcaa59e2700 1 -- 172.21.15.134:0/1839623458 wait complete. 2023-11-16T01:10:55.342 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:53.951719+0000 mgr.smithi133.xdyttl (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:55.339166+0000 mon.smithi133 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.134:0/1839623458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:56.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:56.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 -- 172.21.15.134:0/268533297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 msgr2=0x7f57d00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/268533297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d00f9150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f57b8009fb0 tx=0x7f57b8009b30).stop 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 -- 172.21.15.134:0/268533297 shutdown_connections 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/268533297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d00f9150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 -- 172.21.15.134:0/268533297 >> 172.21.15.134:0/268533297 conn(0x7f57d00f0b70 msgr2=0x7f57d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 -- 172.21.15.134:0/268533297 shutdown_connections 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.965+0000 7f57d5f9d700 1 -- 172.21.15.134:0/268533297 wait complete. 2023-11-16T01:10:57.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.966+0000 7f57d5f9d700 1 Processor -- start 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.966+0000 7f57d5f9d700 1 -- start start 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.966+0000 7f57d5f9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.966+0000 7f57d5f9d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0106bd0 con 0x7f57d0060220 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.966+0000 7f57cf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57cf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55074/0 (socket says 172.21.15.134:55074) 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57cf7fe700 1 -- 172.21.15.134:0/3482687918 learned_addr learned my addr 172.21.15.134:0/3482687918 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57cf7fe700 1 -- 172.21.15.134:0/3482687918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8005040 con 0x7f57d0060220 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57cf7fe700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f57b8013510 tx=0x7f57b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b8009320 con 0x7f57d0060220 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d0106dd0 con 0x7f57d0060220 2023-11-16T01:10:57.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57b8013f00 con 0x7f57d0060220 2023-11-16T01:10:57.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b8014dc0 con 0x7f57d0060220 2023-11-16T01:10:57.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.967+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d0107260 con 0x7f57d0060220 2023-11-16T01:10:57.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.968+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57b801d3d0 con 0x7f57d0060220 2023-11-16T01:10:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.968+0000 7f57c7fff700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:10:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.968+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57b8041100 con 0x7f57d0060220 2023-11-16T01:10:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.968+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d0045ad0 con 0x7f57d0060220 2023-11-16T01:10:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.968+0000 7f57ceffd700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:10:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.969+0000 7f57ceffd700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 0x7f57bc032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f57c00096a0 tx=0x7f57c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:10:57.974 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:57.973+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57b8012920 con 0x7f57d0060220 2023-11-16T01:10:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:55.952092+0000 mgr.smithi133.xdyttl (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:10:58.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.179+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57d0107fb0 con 0x7f57d0060220 2023-11-16T01:10:58.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.179+0000 7f57c7fff700 1 -- 172.21.15.134:0/3482687918 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57b8012330 con 0x7f57d0060220 2023-11-16T01:10:58.182 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:10:58.182 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:10:58.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 msgr2=0x7f57bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 0x7f57bc032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f57c00096a0 tx=0x7f57c0006b40).stop 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 msgr2=0x7f57d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f57b8013510 tx=0x7f57b800b040).stop 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 shutdown_connections 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:58.184 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 --2- 172.21.15.134:0/3482687918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57d0060220 0x7f57d01066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:10:58.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 >> 172.21.15.134:0/3482687918 conn(0x7f57d00f0b70 msgr2=0x7f57d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:10:58.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 shutdown_connections 2023-11-16T01:10:58.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:10:58.181+0000 7f57d5f9d700 1 -- 172.21.15.134:0/3482687918 wait complete. 2023-11-16T01:10:58.185 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:10:59.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:58 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:10:58.180983+0000 mon.smithi133 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.134:0/3482687918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:10:59.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:10:59.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:10:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:57.952474+0000 mgr.smithi133.xdyttl (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 -- 172.21.15.134:0/1265261285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 msgr2=0x7feef80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 --2- 172.21.15.134:0/1265261285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef80f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7feef0009960 tx=0x7feef0004d10).stop 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 -- 172.21.15.134:0/1265261285 shutdown_connections 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 --2- 172.21.15.134:0/1265261285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef80f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 -- 172.21.15.134:0/1265261285 >> 172.21.15.134:0/1265261285 conn(0x7feef80f0b70 msgr2=0x7feef80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 -- 172.21.15.134:0/1265261285 shutdown_connections 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.780+0000 7fef00421700 1 -- 172.21.15.134:0/1265261285 wait complete. 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7fef00421700 1 Processor -- start 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7fef00421700 1 -- start start 2023-11-16T01:11:00.783 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7fef00421700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:00.784 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7fef00421700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef8105b20 con 0x7feef80f5410 2023-11-16T01:11:00.784 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7feefe1bd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:00.784 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7feefe1bd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34396/0 (socket says 172.21.15.134:34396) 2023-11-16T01:11:00.784 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.781+0000 7feefe1bd700 1 -- 172.21.15.134:0/767239086 learned_addr learned my addr 172.21.15.134:0/767239086 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:00.785 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7feefe1bd700 1 -- 172.21.15.134:0/767239086 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0009640 con 0x7feef80f5410 2023-11-16T01:11:00.785 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7feefe1bd700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feef0004b70 tx=0x7feef0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:00.785 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef0018070 con 0x7feef80f5410 2023-11-16T01:11:00.785 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef8105d20 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feef000b870 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef001c560 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.782+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef81061b0 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.783+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feef0013630 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.783+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef8045ad0 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.783+0000 7feeeeffd700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 0x7feee4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.783+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feef0041290 con 0x7feef80f5410 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.783+0000 7feefd9bc700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 0x7feee4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.784+0000 7feefd9bc700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 0x7feee4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:00.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.788+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feef0004810 con 0x7feef80f5410 2023-11-16T01:11:00.995 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.993+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feef80fae60 con 0x7feef80f5410 2023-11-16T01:11:00.995 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.994+0000 7feeeeffd700 1 -- 172.21.15.134:0/767239086 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feef0012380 con 0x7feef80f5410 2023-11-16T01:11:00.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:00.995 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.995+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 msgr2=0x7feee4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 0x7feee4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).stop 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 msgr2=0x7feef8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feef0004b70 tx=0x7feef0004d10).stop 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 shutdown_connections 2023-11-16T01:11:00.997 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feee402ff60 0x7feee4032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 --2- 172.21.15.134:0/767239086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feef80f5410 0x7feef8105630 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 >> 172.21.15.134:0/767239086 conn(0x7feef80f0b70 msgr2=0x7feef80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 shutdown_connections 2023-11-16T01:11:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:00.996+0000 7fef00421700 1 -- 172.21.15.134:0/767239086 wait complete. 2023-11-16T01:11:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:10:59.952812+0000 mgr.smithi133.xdyttl (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:00.995415+0000 mon.smithi133 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.134:0/767239086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:02.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:02.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:03.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 -- 172.21.15.134:0/3282299255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 msgr2=0x7f67bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 --2- 172.21.15.134:0/3282299255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc0f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f67b0009fb0 tx=0x7f67b0009b30).stop 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 -- 172.21.15.134:0/3282299255 shutdown_connections 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 --2- 172.21.15.134:0/3282299255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc0f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 -- 172.21.15.134:0/3282299255 >> 172.21.15.134:0/3282299255 conn(0x7f67bc0f1c70 msgr2=0x7f67bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 -- 172.21.15.134:0/3282299255 shutdown_connections 2023-11-16T01:11:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 -- 172.21.15.134:0/3282299255 wait complete. 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.544+0000 7f67c1db5700 1 Processor -- start 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67c1db5700 1 -- start start 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67c1db5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67c1db5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc106c20 con 0x7f67bc0f6510 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67bb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67bb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34408/0 (socket says 172.21.15.134:34408) 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.545+0000 7f67bb7fe700 1 -- 172.21.15.134:0/1772866451 learned_addr learned my addr 172.21.15.134:0/1772866451 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67bb7fe700 1 -- 172.21.15.134:0/1772866451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b0005040 con 0x7f67bc0f6510 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67bb7fe700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f67b0013510 tx=0x7f67b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b0009320 con 0x7f67bc0f6510 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67b0013f00 con 0x7f67bc0f6510 2023-11-16T01:11:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc106e20 con 0x7f67bc0f6510 2023-11-16T01:11:03.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b0014d90 con 0x7f67bc0f6510 2023-11-16T01:11:03.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.546+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc1072b0 con 0x7f67bc0f6510 2023-11-16T01:11:03.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.547+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67b00263a0 con 0x7f67bc0f6510 2023-11-16T01:11:03.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.547+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc045ad0 con 0x7f67bc0f6510 2023-11-16T01:11:03.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.547+0000 7f67a3fff700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 0x7f67a4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:03.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.547+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67b00410b0 con 0x7f67bc0f6510 2023-11-16T01:11:03.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.547+0000 7f67baffd700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 0x7f67a4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:03.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.548+0000 7f67baffd700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 0x7f67a4032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f67ac0096a0 tx=0x7f67ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:03.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.552+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67b006d2b0 con 0x7f67bc0f6510 2023-11-16T01:11:03.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.759+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67bc0fbf60 con 0x7f67bc0f6510 2023-11-16T01:11:03.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.760+0000 7f67a3fff700 1 -- 172.21.15.134:0/1772866451 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67b00095f0 con 0x7f67bc0f6510 2023-11-16T01:11:03.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:03.762 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.761+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 msgr2=0x7f67a4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.761+0000 7f67c1db5700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 0x7f67a4032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f67ac0096a0 tx=0x7f67ac006b40).stop 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 msgr2=0x7f67bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f67b0013510 tx=0x7f67b000b040).stop 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 shutdown_connections 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f67a402ffb0 0x7f67a4032470 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:03.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 --2- 172.21.15.134:0/1772866451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f67bc0f6510 0x7f67bc106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:03.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 >> 172.21.15.134:0/1772866451 conn(0x7f67bc0f1c70 msgr2=0x7f67bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:03.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 shutdown_connections 2023-11-16T01:11:03.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:03.762+0000 7f67c1db5700 1 -- 172.21.15.134:0/1772866451 wait complete. 2023-11-16T01:11:03.764 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:01.953144+0000 mgr.smithi133.xdyttl (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:03.761615+0000 mon.smithi133 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.134:0/1772866451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:05.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:05.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:03.953451+0000 mgr.smithi133.xdyttl (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/3462711839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 msgr2=0x7f9b9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/3462711839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c0f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9b8c009fb0 tx=0x7f9b8c009b30).stop 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/3462711839 shutdown_connections 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/3462711839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c0f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/3462711839 >> 172.21.15.134:0/3462711839 conn(0x7f9b9c0f1c70 msgr2=0x7f9b9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/3462711839 shutdown_connections 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.352+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/3462711839 wait complete. 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9ba1e1d700 1 Processor -- start 2023-11-16T01:11:06.354 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9ba1e1d700 1 -- start start 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9ba1e1d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9ba1e1d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c106c20 con 0x7f9b9c0f6510 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34432/0 (socket says 172.21.15.134:34432) 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.353+0000 7f9b9b7fe700 1 -- 172.21.15.134:0/949340666 learned_addr learned my addr 172.21.15.134:0/949340666 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9b9b7fe700 1 -- 172.21.15.134:0/949340666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c005040 con 0x7f9b9c0f6510 2023-11-16T01:11:06.355 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9b9b7fe700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9b8c013510 tx=0x7f9b8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c009320 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b8c013f00 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c106e20 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c014d90 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.354+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c1072b0 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.355+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b8c0263a0 con 0x7f9b9c0f6510 2023-11-16T01:11:06.356 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.355+0000 7f9b93fff700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:06.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.355+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b8c041000 con 0x7f9b9c0f6510 2023-11-16T01:11:06.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.355+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c045ad0 con 0x7f9b9c0f6510 2023-11-16T01:11:06.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.355+0000 7f9b9affd700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:06.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.356+0000 7f9b9affd700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 0x7f9b88032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:06.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.360+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b8c012920 con 0x7f9b9c0f6510 2023-11-16T01:11:06.565 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.565+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b9c0fc050 con 0x7f9b9c0f6510 2023-11-16T01:11:06.566 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.565+0000 7f9b93fff700 1 -- 172.21.15.134:0/949340666 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b8c02b020 con 0x7f9b9c0f6510 2023-11-16T01:11:06.566 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:06.566 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 msgr2=0x7f9b88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 0x7f9b88032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).stop 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 msgr2=0x7f9b9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9b8c013510 tx=0x7f9b8c00b040).stop 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 shutdown_connections 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 --2- 172.21.15.134:0/949340666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b9c0f6510 0x7f9b9c106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 >> 172.21.15.134:0/949340666 conn(0x7f9b9c0f1c70 msgr2=0x7f9b9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 shutdown_connections 2023-11-16T01:11:06.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:06.567+0000 7f9ba1e1d700 1 -- 172.21.15.134:0/949340666 wait complete. 2023-11-16T01:11:06.569 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:07.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:06.566105+0000 mon.smithi133 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.134:0/949340666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:07.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:07.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:08.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:05.953695+0000 mgr.smithi133.xdyttl (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- 172.21.15.134:0/1449925553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 msgr2=0x7f6a980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1449925553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a980f66c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f6a88009fb0 tx=0x7f6a88009b30).stop 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- 172.21.15.134:0/1449925553 shutdown_connections 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1449925553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a980f66c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- 172.21.15.134:0/1449925553 >> 172.21.15.134:0/1449925553 conn(0x7f6a980f1ac0 msgr2=0x7f6a980f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- 172.21.15.134:0/1449925553 shutdown_connections 2023-11-16T01:11:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- 172.21.15.134:0/1449925553 wait complete. 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 Processor -- start 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.113+0000 7f6a97fff700 1 -- start start 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a97fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a97fff700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98106990 con 0x7f6a980f62a0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a96ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a96ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34444/0 (socket says 172.21.15.134:34444) 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a96ffd700 1 -- 172.21.15.134:0/1788701043 learned_addr learned my addr 172.21.15.134:0/1788701043 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a96ffd700 1 -- 172.21.15.134:0/1788701043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88005040 con 0x7f6a980f62a0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a96ffd700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6a88009830 tx=0x7f6a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a88009320 con 0x7f6a980f62a0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a98106b90 con 0x7f6a980f62a0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a880136c0 con 0x7f6a980f62a0 2023-11-16T01:11:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a8801ee90 con 0x7f6a980f62a0 2023-11-16T01:11:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.114+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a98107020 con 0x7f6a980f62a0 2023-11-16T01:11:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.115+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a8801e420 con 0x7f6a980f62a0 2023-11-16T01:11:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.115+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a98045ad0 con 0x7f6a980f62a0 2023-11-16T01:11:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.115+0000 7f6a7f7fe700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.115+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a88040fc0 con 0x7f6a980f62a0 2023-11-16T01:11:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.115+0000 7f6a967fc700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.116+0000 7f6a967fc700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 0x7f6a800323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.120+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a8806d2b0 con 0x7f6a980f62a0 2023-11-16T01:11:09.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.323+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a98107e70 con 0x7f6a980f62a0 2023-11-16T01:11:09.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.324+0000 7f6a7f7fe700 1 -- 172.21.15.134:0/1788701043 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a8802c2d0 con 0x7f6a980f62a0 2023-11-16T01:11:09.325 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:09.325 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 msgr2=0x7f6a800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 0x7f6a800323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).stop 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 msgr2=0x7f6a981064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6a88009830 tx=0x7f6a8800b040).stop 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 shutdown_connections 2023-11-16T01:11:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:09.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.325+0000 7f6a97fff700 1 --2- 172.21.15.134:0/1788701043 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a980f62a0 0x7f6a981064a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:09.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.326+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 >> 172.21.15.134:0/1788701043 conn(0x7f6a980f1ac0 msgr2=0x7f6a980f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:09.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.326+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 shutdown_connections 2023-11-16T01:11:09.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:09.326+0000 7f6a97fff700 1 -- 172.21.15.134:0/1788701043 wait complete. 2023-11-16T01:11:09.327 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:07.954022+0000 mgr.smithi133.xdyttl (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:09.324300+0000 mon.smithi133 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.134:0/1788701043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:10.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:10.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:11.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 -- 172.21.15.134:0/2689610230 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 msgr2=0x7fe0080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 --2- 172.21.15.134:0/2689610230 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe0080f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fdff8009fb0 tx=0x7fdff8009b30).stop 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 -- 172.21.15.134:0/2689610230 shutdown_connections 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 --2- 172.21.15.134:0/2689610230 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe0080f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 -- 172.21.15.134:0/2689610230 >> 172.21.15.134:0/2689610230 conn(0x7fe0080f1c70 msgr2=0x7fe0080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 -- 172.21.15.134:0/2689610230 shutdown_connections 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.882+0000 7fe00e342700 1 -- 172.21.15.134:0/2689610230 wait complete. 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe00e342700 1 Processor -- start 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe00e342700 1 -- start start 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe00e342700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe00e342700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe008106c20 con 0x7fe0080f6510 2023-11-16T01:11:11.884 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe007fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:11.885 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe007fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56842/0 (socket says 172.21.15.134:56842) 2023-11-16T01:11:11.885 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.883+0000 7fe007fff700 1 -- 172.21.15.134:0/228811977 learned_addr learned my addr 172.21.15.134:0/228811977 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe007fff700 1 -- 172.21.15.134:0/228811977 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff8005040 con 0x7fe0080f6510 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe007fff700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fdff8013510 tx=0x7fdff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff8009320 con 0x7fe0080f6510 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe008106e20 con 0x7fe0080f6510 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdff8013f00 con 0x7fe0080f6510 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff8014d90 con 0x7fe0080f6510 2023-11-16T01:11:11.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.884+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0081072b0 con 0x7fe0080f6510 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdff8014420 con 0x7fe0080f6510 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe008045ad0 con 0x7fe0080f6510 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe004ff9700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 0x7fdff0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdff8041080 con 0x7fe0080f6510 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe0077fe700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 0x7fdff0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:11.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.885+0000 7fe0077fe700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 0x7fdff0032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fdffc0096a0 tx=0x7fdffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:11.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:11.889+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdff80095f0 con 0x7fe0080f6510 2023-11-16T01:11:12.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.095+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0080fbf60 con 0x7fe0080f6510 2023-11-16T01:11:12.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.096+0000 7fe004ff9700 1 -- 172.21.15.134:0/228811977 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdff801db70 con 0x7fe0080f6510 2023-11-16T01:11:12.097 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:12.097 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:12.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 msgr2=0x7fdff0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:12.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 0x7fdff0032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fdffc0096a0 tx=0x7fdffc006b40).stop 2023-11-16T01:11:12.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 msgr2=0x7fe008106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:12.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fdff8013510 tx=0x7fdff800b040).stop 2023-11-16T01:11:12.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 shutdown_connections 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fdff002ff60 0x7fdff0032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.097+0000 7fe00e342700 1 --2- 172.21.15.134:0/228811977 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe0080f6510 0x7fe008106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.098+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 >> 172.21.15.134:0/228811977 conn(0x7fe0080f1c70 msgr2=0x7fe0080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.098+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 shutdown_connections 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:12.098+0000 7fe00e342700 1 -- 172.21.15.134:0/228811977 wait complete. 2023-11-16T01:11:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:12.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:09.954390+0000 mgr.smithi133.xdyttl (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:13.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:12 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:12.096577+0000 mon.smithi133 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.134:0/228811977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:13.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:13.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:14.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:11.954690+0000 mgr.smithi133.xdyttl (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 -- 172.21.15.134:0/1400133137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 msgr2=0x7fba7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 --2- 172.21.15.134:0/1400133137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c0f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fba6c009fb0 tx=0x7fba6c009b30).stop 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 -- 172.21.15.134:0/1400133137 shutdown_connections 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 --2- 172.21.15.134:0/1400133137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c0f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 -- 172.21.15.134:0/1400133137 >> 172.21.15.134:0/1400133137 conn(0x7fba7c0f1c70 msgr2=0x7fba7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 -- 172.21.15.134:0/1400133137 shutdown_connections 2023-11-16T01:11:14.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.661+0000 7fba83991700 1 -- 172.21.15.134:0/1400133137 wait complete. 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba83991700 1 Processor -- start 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba83991700 1 -- start start 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba83991700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba83991700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c106c20 con 0x7fba7c0f6510 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba8172d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba8172d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56868/0 (socket says 172.21.15.134:56868) 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.662+0000 7fba8172d700 1 -- 172.21.15.134:0/3409381453 learned_addr learned my addr 172.21.15.134:0/3409381453 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba8172d700 1 -- 172.21.15.134:0/3409381453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c005040 con 0x7fba7c0f6510 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba8172d700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fba6c013510 tx=0x7fba6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6c009320 con 0x7fba7c0f6510 2023-11-16T01:11:14.664 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c106e20 con 0x7fba7c0f6510 2023-11-16T01:11:14.665 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba6c013f00 con 0x7fba7c0f6510 2023-11-16T01:11:14.665 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6c014d90 con 0x7fba7c0f6510 2023-11-16T01:11:14.665 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.663+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c1072b0 con 0x7fba7c0f6510 2023-11-16T01:11:14.665 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.664+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba6c0095f0 con 0x7fba7c0f6510 2023-11-16T01:11:14.666 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.664+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c045ad0 con 0x7fba7c0f6510 2023-11-16T01:11:14.666 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.664+0000 7fba71ffb700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:14.666 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.664+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba6c0410c0 con 0x7fba7c0f6510 2023-11-16T01:11:14.666 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.664+0000 7fba80f2c700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:14.666 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.665+0000 7fba80f2c700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 0x7fba68032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fba780096a0 tx=0x7fba78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:14.669 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.669+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba6c014420 con 0x7fba7c0f6510 2023-11-16T01:11:14.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.875+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba7c0fbf60 con 0x7fba7c0f6510 2023-11-16T01:11:14.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.876+0000 7fba71ffb700 1 -- 172.21.15.134:0/3409381453 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba6c01d7f0 con 0x7fba7c0f6510 2023-11-16T01:11:14.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:14.877 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:14.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 msgr2=0x7fba68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 0x7fba68032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fba780096a0 tx=0x7fba78006b40).stop 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 msgr2=0x7fba7c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fba6c013510 tx=0x7fba6c00b040).stop 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 shutdown_connections 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 --2- 172.21.15.134:0/3409381453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba7c0f6510 0x7fba7c106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 >> 172.21.15.134:0/3409381453 conn(0x7fba7c0f1c70 msgr2=0x7fba7c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 shutdown_connections 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:14.878+0000 7fba83991700 1 -- 172.21.15.134:0/3409381453 wait complete. 2023-11-16T01:11:14.879 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:16.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:16.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:16.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:13.955031+0000 mgr.smithi133.xdyttl (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:16.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:14.876474+0000 mon.smithi133 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.134:0/3409381453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:17.481 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 -- 172.21.15.134:0/3236539534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 msgr2=0x7f77480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:17.481 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 --2- 172.21.15.134:0/3236539534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480f55c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7738009fb0 tx=0x7f7738009b30).stop 2023-11-16T01:11:17.481 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 -- 172.21.15.134:0/3236539534 shutdown_connections 2023-11-16T01:11:17.481 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 --2- 172.21.15.134:0/3236539534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480f55c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:17.481 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 -- 172.21.15.134:0/3236539534 >> 172.21.15.134:0/3236539534 conn(0x7f77480f09a0 msgr2=0x7f77480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 -- 172.21.15.134:0/3236539534 shutdown_connections 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.470+0000 7f774d59e700 1 -- 172.21.15.134:0/3236539534 wait complete. 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f774d59e700 1 Processor -- start 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f774d59e700 1 -- start start 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f774d59e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f774d59e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77480fb5a0 con 0x7f77480f51a0 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f7747fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:17.482 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f7747fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56882/0 (socket says 172.21.15.134:56882) 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f7747fff700 1 -- 172.21.15.134:0/4024002221 learned_addr learned my addr 172.21.15.134:0/4024002221 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.471+0000 7f7747fff700 1 -- 172.21.15.134:0/4024002221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738005040 con 0x7f77480f51a0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f7747fff700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7738013510 tx=0x7f773800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7738009320 con 0x7f77480f51a0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7738013f00 con 0x7f77480f51a0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77480fb740 con 0x7f77480f51a0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7738014dc0 con 0x7f77480f51a0 2023-11-16T01:11:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.472+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77480fbbd0 con 0x7f77480f51a0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7738014420 con 0x7f77480f51a0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7748059100 con 0x7f77480f51a0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f7744ff9700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7738041050 con 0x7f77480f51a0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f77477fe700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.473+0000 7f77477fe700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:17.484 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.478+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77380095f0 con 0x7f77480f51a0 2023-11-16T01:11:17.683 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.683+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77480fabe0 con 0x7f77480f51a0 2023-11-16T01:11:17.684 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.683+0000 7f7744ff9700 1 -- 172.21.15.134:0/4024002221 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f773801cb10 con 0x7f77480f51a0 2023-11-16T01:11:17.684 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:17.684 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 msgr2=0x7f7734032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).stop 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 msgr2=0x7f77480fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7738013510 tx=0x7f773800b040).stop 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 shutdown_connections 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:17.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 --2- 172.21.15.134:0/4024002221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77480f51a0 0x7f77480fd120 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 >> 172.21.15.134:0/4024002221 conn(0x7f77480f09a0 msgr2=0x7f774805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 shutdown_connections 2023-11-16T01:11:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:17.685+0000 7f774d59e700 1 -- 172.21.15.134:0/4024002221 wait complete. 2023-11-16T01:11:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:15.955411+0000 mgr.smithi133.xdyttl (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:17.684044+0000 mon.smithi133 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.134:0/4024002221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:19.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:19.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:20.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:17.955708+0000 mgr.smithi133.xdyttl (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.318+0000 7fe1be058700 1 -- 172.21.15.134:0/4182503736 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 msgr2=0x7fe1b80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.318+0000 7fe1be058700 1 --2- 172.21.15.134:0/4182503736 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b80f9150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe1a8009fb0 tx=0x7fe1a8009b30).stop 2023-11-16T01:11:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- 172.21.15.134:0/4182503736 shutdown_connections 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 --2- 172.21.15.134:0/4182503736 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b80f9150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- 172.21.15.134:0/4182503736 >> 172.21.15.134:0/4182503736 conn(0x7fe1b80f0b70 msgr2=0x7fe1b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- 172.21.15.134:0/4182503736 shutdown_connections 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- 172.21.15.134:0/4182503736 wait complete. 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 Processor -- start 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- start start 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.319+0000 7fe1be058700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b8106ba0 con 0x7fe1b8060220 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1b77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1b77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41856/0 (socket says 172.21.15.134:41856) 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1b77fe700 1 -- 172.21.15.134:0/3787576585 learned_addr learned my addr 172.21.15.134:0/3787576585 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1b77fe700 1 -- 172.21.15.134:0/3787576585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a8005040 con 0x7fe1b8060220 2023-11-16T01:11:20.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1b77fe700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe1a8013510 tx=0x7fe1a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a8009320 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b8106da0 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1a8013f00 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a8014dc0 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.320+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b8107230 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.321+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1a8014420 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.321+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b8045ad0 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.321+0000 7fe19ffff700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.321+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1a8041050 con 0x7fe1b8060220 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.322+0000 7fe1b6ffd700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:20.323 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.322+0000 7fe1b6ffd700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 0x7fe1a0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe1ac0096a0 tx=0x7fe1ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:20.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.326+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1a80095f0 con 0x7fe1b8060220 2023-11-16T01:11:20.535 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.534+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1b81080e0 con 0x7fe1b8060220 2023-11-16T01:11:20.535 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.535+0000 7fe19ffff700 1 -- 172.21.15.134:0/3787576585 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1a801cb10 con 0x7fe1b8060220 2023-11-16T01:11:20.535 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:20.536 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.536+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 msgr2=0x7fe1a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.536+0000 7fe1be058700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 0x7fe1a0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe1ac0096a0 tx=0x7fe1ac006b40).stop 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 msgr2=0x7fe1b81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe1a8013510 tx=0x7fe1a800b040).stop 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 shutdown_connections 2023-11-16T01:11:20.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe1a002ff60 0x7fe1a0032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:20.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 --2- 172.21.15.134:0/3787576585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1b8060220 0x7fe1b81066b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:20.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 >> 172.21.15.134:0/3787576585 conn(0x7fe1b80f0b70 msgr2=0x7fe1b80f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:20.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 shutdown_connections 2023-11-16T01:11:20.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:20.537+0000 7fe1be058700 1 -- 172.21.15.134:0/3787576585 wait complete. 2023-11-16T01:11:20.538 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:21.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:20.535571+0000 mon.smithi133 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.134:0/3787576585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:21.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:21.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:22.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:19.956023+0000 mgr.smithi133.xdyttl (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:23.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 -- 172.21.15.134:0/1959743385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 msgr2=0x7fa1280fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:23.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 --2- 172.21.15.134:0/1959743385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1280fb550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa110009fb0 tx=0x7fa110009b30).stop 2023-11-16T01:11:23.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 -- 172.21.15.134:0/1959743385 shutdown_connections 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 --2- 172.21.15.134:0/1959743385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1280fb550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 -- 172.21.15.134:0/1959743385 >> 172.21.15.134:0/1959743385 conn(0x7fa1280f0b70 msgr2=0x7fa1280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 -- 172.21.15.134:0/1959743385 shutdown_connections 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.174+0000 7fa12cafe700 1 -- 172.21.15.134:0/1959743385 wait complete. 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.175+0000 7fa12cafe700 1 Processor -- start 2023-11-16T01:11:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.175+0000 7fa12cafe700 1 -- start start 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.175+0000 7fa12cafe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.175+0000 7fa12cafe700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128108db0 con 0x7fa128060220 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41874/0 (socket says 172.21.15.134:41874) 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12659c700 1 -- 172.21.15.134:0/2810794360 learned_addr learned my addr 172.21.15.134:0/2810794360 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:23.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12659c700 1 -- 172.21.15.134:0/2810794360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa110005040 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12659c700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa110013510 tx=0x7fa11000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa110009320 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa110013f00 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa128108fb0 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa110014dc0 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.176+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa128109440 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.177+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa128045ad0 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.177+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa110014420 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.177+0000 7fa11effd700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.177+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa110041020 con 0x7fa128060220 2023-11-16T01:11:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.177+0000 7fa125d9b700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:23.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.178+0000 7fa125d9b700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 0x7fa114032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa1180096a0 tx=0x7fa118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:23.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.182+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1100095f0 con 0x7fa128060220 2023-11-16T01:11:23.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.388+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa12810a290 con 0x7fa128060220 2023-11-16T01:11:23.390 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.388+0000 7fa11effd700 1 -- 172.21.15.134:0/2810794360 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa11001cb10 con 0x7fa128060220 2023-11-16T01:11:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:23.390 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:23.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 msgr2=0x7fa114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 0x7fa114032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa1180096a0 tx=0x7fa118006b40).stop 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 msgr2=0x7fa1281088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa110013510 tx=0x7fa11000b040).stop 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 shutdown_connections 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 --2- 172.21.15.134:0/2810794360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa128060220 0x7fa1281088c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 >> 172.21.15.134:0/2810794360 conn(0x7fa1280f0b70 msgr2=0x7fa1280f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 shutdown_connections 2023-11-16T01:11:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:23.390+0000 7fa12cafe700 1 -- 172.21.15.134:0/2810794360 wait complete. 2023-11-16T01:11:23.393 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:24.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:21.956369+0000 mgr.smithi133.xdyttl (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:24.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:23.389943+0000 mon.smithi133 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.134:0/2810794360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:24.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:24.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:26.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 -- 172.21.15.134:0/3121614141 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 msgr2=0x7fe12c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 --2- 172.21.15.134:0/3121614141 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe11c009fb0 tx=0x7fe11c009b30).stop 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 -- 172.21.15.134:0/3121614141 shutdown_connections 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 --2- 172.21.15.134:0/3121614141 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 -- 172.21.15.134:0/3121614141 >> 172.21.15.134:0/3121614141 conn(0x7fe12c0f0b70 msgr2=0x7fe12c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 -- 172.21.15.134:0/3121614141 shutdown_connections 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.036+0000 7fe130e39700 1 -- 172.21.15.134:0/3121614141 wait complete. 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe130e39700 1 Processor -- start 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe130e39700 1 -- start start 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe130e39700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe130e39700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c060240 con 0x7fe12c0f5410 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe12a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:26.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe12a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41880/0 (socket says 172.21.15.134:41880) 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.037+0000 7fe12a59c700 1 -- 172.21.15.134:0/3247126092 learned_addr learned my addr 172.21.15.134:0/3247126092 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe12a59c700 1 -- 172.21.15.134:0/3247126092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c005040 con 0x7fe12c0f5410 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe12a59c700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe11c013510 tx=0x7fe11c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe11c009320 con 0x7fe12c0f5410 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe11c013f00 con 0x7fe12c0f5410 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c060440 con 0x7fe12c0f5410 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe11c014d90 con 0x7fe12c0f5410 2023-11-16T01:11:26.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.038+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c109ae0 con 0x7fe12c0f5410 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe11c014420 con 0x7fe12c0f5410 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c045ad0 con 0x7fe12c0f5410 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe122ffd700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe11c0410c0 con 0x7fe12c0f5410 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe129d9b700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.039+0000 7fe129d9b700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 0x7fe118032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe1140096a0 tx=0x7fe114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:26.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.043+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe11c0095f0 con 0x7fe12c0f5410 2023-11-16T01:11:26.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.250+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe12c0fae60 con 0x7fe12c0f5410 2023-11-16T01:11:26.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.251+0000 7fe122ffd700 1 -- 172.21.15.134:0/3247126092 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe11c01db70 con 0x7fe12c0f5410 2023-11-16T01:11:26.252 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:26.252 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 msgr2=0x7fe118032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 0x7fe118032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe1140096a0 tx=0x7fe114006b40).stop 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 msgr2=0x7fe12c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe11c013510 tx=0x7fe11c00b040).stop 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 shutdown_connections 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 --2- 172.21.15.134:0/3247126092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe12c0f5410 0x7fe12c109430 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 >> 172.21.15.134:0/3247126092 conn(0x7fe12c0f0b70 msgr2=0x7fe12c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 shutdown_connections 2023-11-16T01:11:26.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:26.252+0000 7fe130e39700 1 -- 172.21.15.134:0/3247126092 wait complete. 2023-11-16T01:11:26.255 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:26.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:23.956695+0000 mgr.smithi133.xdyttl (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:27.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:26 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:26.252099+0000 mon.smithi133 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.134:0/3247126092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:27.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:27.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:28.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:25.956962+0000 mgr.smithi133.xdyttl (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:28.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 -- 172.21.15.134:0/185799553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 msgr2=0x7ff7740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 --2- 172.21.15.134:0/185799553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff7740f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ff764009fb0 tx=0x7ff764009b30).stop 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 -- 172.21.15.134:0/185799553 shutdown_connections 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 --2- 172.21.15.134:0/185799553 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff7740f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 -- 172.21.15.134:0/185799553 >> 172.21.15.134:0/185799553 conn(0x7ff7740f0b70 msgr2=0x7ff7740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 -- 172.21.15.134:0/185799553 shutdown_connections 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 -- 172.21.15.134:0/185799553 wait complete. 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.846+0000 7ff77a858700 1 Processor -- start 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff77a858700 1 -- start start 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff77a858700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff77a858700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774106c50 con 0x7ff7740f5410 2023-11-16T01:11:28.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff773fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff773fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41900/0 (socket says 172.21.15.134:41900) 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff773fff700 1 -- 172.21.15.134:0/3333334720 learned_addr learned my addr 172.21.15.134:0/3333334720 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff773fff700 1 -- 172.21.15.134:0/3333334720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff764005040 con 0x7ff7740f5410 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.847+0000 7ff773fff700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff764009f80 tx=0x7ff76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.848+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff764009320 con 0x7ff7740f5410 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.848+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff774106e50 con 0x7ff7740f5410 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.848+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff764013ec0 con 0x7ff7740f5410 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.848+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76401ece0 con 0x7ff7740f5410 2023-11-16T01:11:28.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.848+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7741072e0 con 0x7ff7740f5410 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff76401e420 con 0x7ff7740f5410 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff774045ad0 con 0x7ff7740f5410 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff770ff9700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff764040d90 con 0x7ff7740f5410 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff7737fe700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.849+0000 7ff7737fe700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 0x7ff75c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7680096a0 tx=0x7ff768006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:28.855 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:28.854+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff76401e730 con 0x7ff7740f5410 2023-11-16T01:11:29.058 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.057+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff774108130 con 0x7ff7740f5410 2023-11-16T01:11:29.059 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.058+0000 7ff770ff9700 1 -- 172.21.15.134:0/3333334720 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff76401cb10 con 0x7ff7740f5410 2023-11-16T01:11:29.059 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:29.059 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 msgr2=0x7ff75c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 0x7ff75c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7680096a0 tx=0x7ff768006b40).stop 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 msgr2=0x7ff774106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff764009f80 tx=0x7ff76400b040).stop 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 shutdown_connections 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 --2- 172.21.15.134:0/3333334720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7740f5410 0x7ff774106760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:29.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 >> 172.21.15.134:0/3333334720 conn(0x7ff7740f0b70 msgr2=0x7ff77405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 shutdown_connections 2023-11-16T01:11:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:29.059+0000 7ff77a858700 1 -- 172.21.15.134:0/3333334720 wait complete. 2023-11-16T01:11:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:27.957269+0000 mgr.smithi133.xdyttl (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:29.059243+0000 mon.smithi133 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.134:0/3333334720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:30.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:30.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:30.072974+0000 mon.smithi133 (mon.0) 173 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:11:31.323 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:30.074404+0000 mon.smithi133 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:11:31.632 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 -- 172.21.15.134:0/1555069986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 msgr2=0x7f68680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 --2- 172.21.15.134:0/1555069986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6858009fb0 tx=0x7f6858009b30).stop 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 -- 172.21.15.134:0/1555069986 shutdown_connections 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 --2- 172.21.15.134:0/1555069986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 -- 172.21.15.134:0/1555069986 >> 172.21.15.134:0/1555069986 conn(0x7f68680f0b70 msgr2=0x7f68680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 -- 172.21.15.134:0/1555069986 shutdown_connections 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.630+0000 7f686e8b6700 1 -- 172.21.15.134:0/1555069986 wait complete. 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f686e8b6700 1 Processor -- start 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f686e8b6700 1 -- start start 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f686e8b6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:31.633 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f686e8b6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68680fb820 con 0x7f68680f5410 2023-11-16T01:11:31.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f6867fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:31.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f6867fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54384/0 (socket says 172.21.15.134:54384) 2023-11-16T01:11:31.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f6867fff700 1 -- 172.21.15.134:0/587817563 learned_addr learned my addr 172.21.15.134:0/587817563 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:31.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.631+0000 7f6867fff700 1 -- 172.21.15.134:0/587817563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6858005040 con 0x7f68680f5410 2023-11-16T01:11:31.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f6867fff700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6858014000 tx=0x7f685800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:31.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68580095f0 con 0x7f68680f5410 2023-11-16T01:11:31.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68680fb9c0 con 0x7f68680f5410 2023-11-16T01:11:31.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6858007e70 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6858014ea0 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.632+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68680fbe50 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.633+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6858025070 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.633+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868059100 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.633+0000 7f6864ff9700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 0x7f6854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.633+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6858040930 con 0x7f68680f5410 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.633+0000 7f68677fe700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 0x7f6854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:31.636 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.634+0000 7f68677fe700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 0x7f6854032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:31.639 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.638+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f685800f4e0 con 0x7f68680f5410 2023-11-16T01:11:31.849 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.847+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68680fae60 con 0x7f68680f5410 2023-11-16T01:11:31.849 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.848+0000 7f6864ff9700 1 -- 172.21.15.134:0/587817563 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f685800f300 con 0x7f68680f5410 2023-11-16T01:11:31.850 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:31.850 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.849+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 msgr2=0x7f6854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.849+0000 7f686e8b6700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 0x7f6854032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).stop 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.849+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 msgr2=0x7f68680fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.849+0000 7f686e8b6700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6858014000 tx=0x7f685800b040).stop 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 shutdown_connections 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f685402ff60 0x7f6854032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 --2- 172.21.15.134:0/587817563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68680f5410 0x7f68680fd3a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:31.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 >> 172.21.15.134:0/587817563 conn(0x7f68680f0b70 msgr2=0x7f68680f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:31.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 shutdown_connections 2023-11-16T01:11:31.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:31.850+0000 7f686e8b6700 1 -- 172.21.15.134:0/587817563 wait complete. 2023-11-16T01:11:31.852 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:32.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:29.957615+0000 mgr.smithi133.xdyttl (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:33.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:33.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:33.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:31.849717+0000 mon.smithi133 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.134:0/587817563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:34.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:31.957933+0000 mgr.smithi133.xdyttl (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:34.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 -- 172.21.15.134:0/3645300279 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 msgr2=0x7f51e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:34.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 --2- 172.21.15.134:0/3645300279 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e40f55c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f51d4009fb0 tx=0x7f51d4009b30).stop 2023-11-16T01:11:34.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 -- 172.21.15.134:0/3645300279 shutdown_connections 2023-11-16T01:11:34.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 --2- 172.21.15.134:0/3645300279 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e40f55c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:34.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 -- 172.21.15.134:0/3645300279 >> 172.21.15.134:0/3645300279 conn(0x7f51e40f09a0 msgr2=0x7f51e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:34.535 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 -- 172.21.15.134:0/3645300279 shutdown_connections 2023-11-16T01:11:34.535 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.532+0000 7f51e3fff700 1 -- 172.21.15.134:0/3645300279 wait complete. 2023-11-16T01:11:34.536 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e3fff700 1 Processor -- start 2023-11-16T01:11:34.536 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e3fff700 1 -- start start 2023-11-16T01:11:34.536 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e3fff700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e41069b0 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54406/0 (socket says 172.21.15.134:54406) 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.533+0000 7f51e2ffd700 1 -- 172.21.15.134:0/2855823631 learned_addr learned my addr 172.21.15.134:0/2855823631 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51e2ffd700 1 -- 172.21.15.134:0/2855823631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4005040 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51e2ffd700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f51d4013510 tx=0x7f51d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51d4009320 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51d4013f00 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e4106bb0 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51d4014dc0 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.534+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e4107040 con 0x7f51e40f51a0 2023-11-16T01:11:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.535+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51d401d3d0 con 0x7f51e40f51a0 2023-11-16T01:11:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.535+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e4045ad0 con 0x7f51e40f51a0 2023-11-16T01:11:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.535+0000 7f51cb7fe700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 0x7f51cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.535+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51d4041060 con 0x7f51e40f51a0 2023-11-16T01:11:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.535+0000 7f51e27fc700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 0x7f51cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.536+0000 7f51e27fc700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 0x7f51cc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f51d80096a0 tx=0x7f51d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:34.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.540+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51d40095f0 con 0x7f51e40f51a0 2023-11-16T01:11:34.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.743+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51e4107ef0 con 0x7f51e40f51a0 2023-11-16T01:11:34.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.744+0000 7f51cb7fe700 1 -- 172.21.15.134:0/2855823631 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f51d401cb10 con 0x7f51e40f51a0 2023-11-16T01:11:34.746 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:34.746 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 msgr2=0x7f51cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 0x7f51cc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f51d80096a0 tx=0x7f51d8006b40).stop 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 msgr2=0x7f51e41064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f51d4013510 tx=0x7f51d400b040).stop 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 shutdown_connections 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f51cc02ff60 0x7f51cc032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 --2- 172.21.15.134:0/2855823631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51e40f51a0 0x7f51e41064c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.745+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 >> 172.21.15.134:0/2855823631 conn(0x7f51e40f09a0 msgr2=0x7f51e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:34.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.746+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 shutdown_connections 2023-11-16T01:11:34.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:34.746+0000 7f51e3fff700 1 -- 172.21.15.134:0/2855823631 wait complete. 2023-11-16T01:11:34.748 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:35.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:34 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:34.745429+0000 mon.smithi133 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.134:0/2855823631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:36.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:36.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:36.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:33.958210+0000 mgr.smithi133.xdyttl (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2800145180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 msgr2=0x7f27d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2800145180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d40f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f27c8009fb0 tx=0x7f27c8009b30).stop 2023-11-16T01:11:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2800145180 shutdown_connections 2023-11-16T01:11:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2800145180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d40f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2800145180 >> 172.21.15.134:0/2800145180 conn(0x7f27d40f0b70 msgr2=0x7f27d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:37.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2800145180 shutdown_connections 2023-11-16T01:11:37.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.432+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2800145180 wait complete. 2023-11-16T01:11:37.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d9a7e700 1 Processor -- start 2023-11-16T01:11:37.439 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d9a7e700 1 -- start start 2023-11-16T01:11:37.439 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d9a7e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d9a7e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4060240 con 0x7f27d40f5410 2023-11-16T01:11:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54428/0 (socket says 172.21.15.134:54428) 2023-11-16T01:11:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.433+0000 7f27d37fe700 1 -- 172.21.15.134:0/2991739751 learned_addr learned my addr 172.21.15.134:0/2991739751 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27d37fe700 1 -- 172.21.15.134:0/2991739751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c8005040 con 0x7f27d40f5410 2023-11-16T01:11:37.441 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27d37fe700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f27c8013510 tx=0x7f27c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:37.441 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c8009320 con 0x7f27d40f5410 2023-11-16T01:11:37.441 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27c8013f00 con 0x7f27d40f5410 2023-11-16T01:11:37.441 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d4060440 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c8014d90 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.434+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d4109ae0 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.435+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27c8014420 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.435+0000 7f27bbfff700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 0x7f27bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.435+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27c80410c0 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.435+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d4045ad0 con 0x7f27d40f5410 2023-11-16T01:11:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.435+0000 7f27d2ffd700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 0x7f27bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:37.443 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.436+0000 7f27d2ffd700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 0x7f27bc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f27c40096a0 tx=0x7f27c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:37.443 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.440+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27c80095f0 con 0x7f27d40f5410 2023-11-16T01:11:37.647 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.645+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27d410a1f0 con 0x7f27d40f5410 2023-11-16T01:11:37.647 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.646+0000 7f27bbfff700 1 -- 172.21.15.134:0/2991739751 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f27c801db70 con 0x7f27d40f5410 2023-11-16T01:11:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:37.648 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.647+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 msgr2=0x7f27bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.647+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 0x7f27bc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f27c40096a0 tx=0x7f27c4006b40).stop 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.647+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 msgr2=0x7f27d4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.647+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f27c8013510 tx=0x7f27c800b040).stop 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 shutdown_connections 2023-11-16T01:11:37.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f27bc02ff60 0x7f27bc032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:37.651 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 --2- 172.21.15.134:0/2991739751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d40f5410 0x7f27d4109430 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:37.651 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 >> 172.21.15.134:0/2991739751 conn(0x7f27d40f0b70 msgr2=0x7f27d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:37.651 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 shutdown_connections 2023-11-16T01:11:37.651 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:37.648+0000 7f27d9a7e700 1 -- 172.21.15.134:0/2991739751 wait complete. 2023-11-16T01:11:37.651 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:38.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:35.958541+0000 mgr.smithi133.xdyttl (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:38.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:37.647432+0000 mon.smithi133 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.134:0/2991739751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:39.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:39.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 -- 172.21.15.134:0/526677531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 msgr2=0x7fe4940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 --2- 172.21.15.134:0/526677531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe4940f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fe488009fb0 tx=0x7fe488009b30).stop 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 -- 172.21.15.134:0/526677531 shutdown_connections 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 --2- 172.21.15.134:0/526677531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe4940f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 -- 172.21.15.134:0/526677531 >> 172.21.15.134:0/526677531 conn(0x7fe4940f0b70 msgr2=0x7fe4940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 -- 172.21.15.134:0/526677531 shutdown_connections 2023-11-16T01:11:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.245+0000 7fe498d8a700 1 -- 172.21.15.134:0/526677531 wait complete. 2023-11-16T01:11:40.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe498d8a700 1 Processor -- start 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe498d8a700 1 -- start start 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe498d8a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe498d8a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe494105b20 con 0x7fe4940f5410 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe49259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe49259c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36434/0 (socket says 172.21.15.134:36434) 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.246+0000 7fe49259c700 1 -- 172.21.15.134:0/4279782354 learned_addr learned my addr 172.21.15.134:0/4279782354 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe49259c700 1 -- 172.21.15.134:0/4279782354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe488005040 con 0x7fe4940f5410 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe49259c700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe488013510 tx=0x7fe48800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe488009320 con 0x7fe4940f5410 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe488013f00 con 0x7fe4940f5410 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe494105d20 con 0x7fe4940f5410 2023-11-16T01:11:40.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe488014d90 con 0x7fe4940f5410 2023-11-16T01:11:40.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.247+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4941061b0 con 0x7fe4940f5410 2023-11-16T01:11:40.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4880095f0 con 0x7fe4940f5410 2023-11-16T01:11:40.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe494045ad0 con 0x7fe4940f5410 2023-11-16T01:11:40.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe482ffd700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 0x7fe47c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:40.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4880410c0 con 0x7fe4940f5410 2023-11-16T01:11:40.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe491d9b700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 0x7fe47c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:40.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.248+0000 7fe491d9b700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 0x7fe47c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe4840096a0 tx=0x7fe484006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:40.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.252+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe488014420 con 0x7fe4940f5410 2023-11-16T01:11:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:37.958901+0000 mgr.smithi133.xdyttl (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:40.457 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.456+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4940fae60 con 0x7fe4940f5410 2023-11-16T01:11:40.457 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.456+0000 7fe482ffd700 1 -- 172.21.15.134:0/4279782354 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe48801d7f0 con 0x7fe4940f5410 2023-11-16T01:11:40.458 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:40.458 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:40.459 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 msgr2=0x7fe47c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:40.459 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 0x7fe47c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe4840096a0 tx=0x7fe484006b40).stop 2023-11-16T01:11:40.459 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 msgr2=0x7fe494105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe488013510 tx=0x7fe48800b040).stop 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 shutdown_connections 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe47c02ff60 0x7fe47c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 --2- 172.21.15.134:0/4279782354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4940f5410 0x7fe494105630 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 >> 172.21.15.134:0/4279782354 conn(0x7fe4940f0b70 msgr2=0x7fe4940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 shutdown_connections 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:40.458+0000 7fe498d8a700 1 -- 172.21.15.134:0/4279782354 wait complete. 2023-11-16T01:11:40.460 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:41.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:40.457781+0000 mon.smithi133 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.134:0/4279782354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:41.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:41.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:42.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:39.959239+0000 mgr.smithi133.xdyttl (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 -- 172.21.15.134:0/1421999001 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 msgr2=0x7f1af00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 --2- 172.21.15.134:0/1421999001 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af00f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1ae4009fb0 tx=0x7f1ae4009b30).stop 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 -- 172.21.15.134:0/1421999001 shutdown_connections 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 --2- 172.21.15.134:0/1421999001 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af00f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 -- 172.21.15.134:0/1421999001 >> 172.21.15.134:0/1421999001 conn(0x7f1af00f0b70 msgr2=0x7f1af00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 -- 172.21.15.134:0/1421999001 shutdown_connections 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.003+0000 7f1af6c41700 1 -- 172.21.15.134:0/1421999001 wait complete. 2023-11-16T01:11:43.006 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.004+0000 7f1af6c41700 1 Processor -- start 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.004+0000 7f1af6c41700 1 -- start start 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.004+0000 7f1af6c41700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.004+0000 7f1af6c41700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0106c20 con 0x7f1af00f5410 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af49dd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af49dd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36444/0 (socket says 172.21.15.134:36444) 2023-11-16T01:11:43.007 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af49dd700 1 -- 172.21.15.134:0/48067391 learned_addr learned my addr 172.21.15.134:0/48067391 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:43.008 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af49dd700 1 -- 172.21.15.134:0/48067391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4005040 con 0x7f1af00f5410 2023-11-16T01:11:43.008 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af49dd700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1ae4013510 tx=0x7f1ae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:43.008 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ae4009320 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ae4013f00 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af0106e20 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ae4014d90 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.005+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af01072b0 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.006+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ae40263a0 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.006+0000 7f1aed7fa700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 0x7f1ae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.006+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af0045ad0 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.006+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ae4040fb0 con 0x7f1af00f5410 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.006+0000 7f1aeffff700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 0x7f1ae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:43.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.007+0000 7f1aeffff700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 0x7f1ae0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f1adc0096a0 tx=0x7f1adc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.011+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ae40095f0 con 0x7f1af00f5410 2023-11-16T01:11:43.219 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.217+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1af00fc080 con 0x7f1af00f5410 2023-11-16T01:11:43.219 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.218+0000 7f1aed7fa700 1 -- 172.21.15.134:0/48067391 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ae402b020 con 0x7f1af00f5410 2023-11-16T01:11:43.219 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:43.220 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.219+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 msgr2=0x7f1ae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.219+0000 7f1af6c41700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 0x7f1ae0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f1adc0096a0 tx=0x7f1adc006b40).stop 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.219+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 msgr2=0x7f1af0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.219+0000 7f1af6c41700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1ae4013510 tx=0x7f1ae400b040).stop 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 shutdown_connections 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1ae002ff60 0x7f1ae0032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 --2- 172.21.15.134:0/48067391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1af00f5410 0x7f1af0106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 >> 172.21.15.134:0/48067391 conn(0x7f1af00f0b70 msgr2=0x7f1af00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:43.222 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 shutdown_connections 2023-11-16T01:11:43.222 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:43.220+0000 7f1af6c41700 1 -- 172.21.15.134:0/48067391 wait complete. 2023-11-16T01:11:43.222 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:41.959654+0000 mgr.smithi133.xdyttl (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:43.219577+0000 mon.smithi133 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.134:0/48067391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:44.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:44.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 -- 172.21.15.134:0/1694994331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 msgr2=0x7f88240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 --2- 172.21.15.134:0/1694994331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f88240f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8818009fb0 tx=0x7f8818009b30).stop 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 -- 172.21.15.134:0/1694994331 shutdown_connections 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 --2- 172.21.15.134:0/1694994331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f88240f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 -- 172.21.15.134:0/1694994331 >> 172.21.15.134:0/1694994331 conn(0x7f88240f0b70 msgr2=0x7f88240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 -- 172.21.15.134:0/1694994331 shutdown_connections 2023-11-16T01:11:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.795+0000 7f882bb5c700 1 -- 172.21.15.134:0/1694994331 wait complete. 2023-11-16T01:11:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f882bb5c700 1 Processor -- start 2023-11-16T01:11:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f882bb5c700 1 -- start start 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f882bb5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f882bb5c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824060220 con 0x7f88240f5410 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f88298f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f88298f8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36462/0 (socket says 172.21.15.134:36462) 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.796+0000 7f88298f8700 1 -- 172.21.15.134:0/654494050 learned_addr learned my addr 172.21.15.134:0/654494050 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f88298f8700 1 -- 172.21.15.134:0/654494050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8818005040 con 0x7f88240f5410 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f88298f8700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8818013510 tx=0x7f881800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8818009320 con 0x7f88240f5410 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8818013f00 con 0x7f88240f5410 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8824060420 con 0x7f88240f5410 2023-11-16T01:11:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8818014d90 con 0x7f88240f5410 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.797+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8824109b00 con 0x7f88240f5410 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.798+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88180263a0 con 0x7f88240f5410 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.798+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8824059100 con 0x7f88240f5410 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.798+0000 7f88167fc700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 0x7f8810032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.798+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88180412f0 con 0x7f88240f5410 2023-11-16T01:11:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.798+0000 7f88290f7700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 0x7f8810032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:45.802 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.799+0000 7f88290f7700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 0x7f8810032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:45.804 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:45.802+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88180126f0 con 0x7f88240f5410 2023-11-16T01:11:46.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.007+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8824045ad0 con 0x7f88240f5410 2023-11-16T01:11:46.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.008+0000 7f88167fc700 1 -- 172.21.15.134:0/654494050 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8818012330 con 0x7f88240f5410 2023-11-16T01:11:46.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:46.010 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 msgr2=0x7f8810032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 0x7f8810032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).stop 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 msgr2=0x7f8824109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8818013510 tx=0x7f881800b040).stop 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 shutdown_connections 2023-11-16T01:11:46.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f881002ff60 0x7f8810032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:46.013 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 --2- 172.21.15.134:0/654494050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88240f5410 0x7f8824109450 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:46.013 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.009+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 >> 172.21.15.134:0/654494050 conn(0x7f88240f0b70 msgr2=0x7f88240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:46.013 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.010+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 shutdown_connections 2023-11-16T01:11:46.013 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:46.010+0000 7f882bb5c700 1 -- 172.21.15.134:0/654494050 wait complete. 2023-11-16T01:11:46.014 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:43.959996+0000 mgr.smithi133.xdyttl (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:47.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:46 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:46.009635+0000 mon.smithi133 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.134:0/654494050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:47.399 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:47.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:45.960343+0000 mgr.smithi133.xdyttl (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:48.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 -- 172.21.15.134:0/1414909516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 msgr2=0x7fd9dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:48.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/1414909516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc0f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fd9d0009fb0 tx=0x7fd9d0009b30).stop 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 -- 172.21.15.134:0/1414909516 shutdown_connections 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/1414909516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc0f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 -- 172.21.15.134:0/1414909516 >> 172.21.15.134:0/1414909516 conn(0x7fd9dc0f0b70 msgr2=0x7fd9dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 -- 172.21.15.134:0/1414909516 shutdown_connections 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.554+0000 7fd9e2f30700 1 -- 172.21.15.134:0/1414909516 wait complete. 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e2f30700 1 Processor -- start 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e2f30700 1 -- start start 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e2f30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e2f30700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc060240 con 0x7fd9dc0f5410 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e0ccc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e0ccc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:36484/0 (socket says 172.21.15.134:36484) 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.555+0000 7fd9e0ccc700 1 -- 172.21.15.134:0/2486668608 learned_addr learned my addr 172.21.15.134:0/2486668608 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9e0ccc700 1 -- 172.21.15.134:0/2486668608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0005040 con 0x7fd9dc0f5410 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9e0ccc700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d0014000 tx=0x7fd9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d00095f0 con 0x7fd9dc0f5410 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc060440 con 0x7fd9dc0f5410 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9d0007e70 con 0x7fd9dc0f5410 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d0014ea0 con 0x7fd9dc0f5410 2023-11-16T01:11:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.556+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc109ae0 con 0x7fd9dc0f5410 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9d0025070 con 0x7fd9dc0f5410 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9d97fa700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc045ad0 con 0x7fd9dc0f5410 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9d0040a30 con 0x7fd9dc0f5410 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9dbfff700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:48.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.557+0000 7fd9dbfff700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd9c80096a0 tx=0x7fd9c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:48.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.562+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9d00079c0 con 0x7fd9dc0f5410 2023-11-16T01:11:48.771 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.771+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9dc0faf50 con 0x7fd9dc0f5410 2023-11-16T01:11:48.773 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.772+0000 7fd9d97fa700 1 -- 172.21.15.134:0/2486668608 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd9d002d090 con 0x7fd9dc0f5410 2023-11-16T01:11:48.774 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:48.774 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 msgr2=0x7fd9cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd9c80096a0 tx=0x7fd9c8006b40).stop 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 msgr2=0x7fd9dc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d0014000 tx=0x7fd9d000b040).stop 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 shutdown_connections 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 --2- 172.21.15.134:0/2486668608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 >> 172.21.15.134:0/2486668608 conn(0x7fd9dc0f0b70 msgr2=0x7fd9dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 shutdown_connections 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:48.774+0000 7fd9e2f30700 1 -- 172.21.15.134:0/2486668608 wait complete. 2023-11-16T01:11:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:49.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:48.772387+0000 mon.smithi133 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.134:0/2486668608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:50.158 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:50.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:47.960680+0000 mgr.smithi133.xdyttl (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:51.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- 172.21.15.134:0/31455177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 msgr2=0x7f48580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:51.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 --2- 172.21.15.134:0/31455177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48580f66c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4854009fb0 tx=0x7f4854009b30).stop 2023-11-16T01:11:51.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- 172.21.15.134:0/31455177 shutdown_connections 2023-11-16T01:11:51.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 --2- 172.21.15.134:0/31455177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48580f66c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:51.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- 172.21.15.134:0/31455177 >> 172.21.15.134:0/31455177 conn(0x7f48580f1aa0 msgr2=0x7f48580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- 172.21.15.134:0/31455177 shutdown_connections 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- 172.21.15.134:0/31455177 wait complete. 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 Processor -- start 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- start start 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.320+0000 7f485f42e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48581069a0 con 0x7f48580f62a0 2023-11-16T01:11:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485e42c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485e42c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42842/0 (socket says 172.21.15.134:42842) 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485e42c700 1 -- 172.21.15.134:0/3523110470 learned_addr learned my addr 172.21.15.134:0/3523110470 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485e42c700 1 -- 172.21.15.134:0/3523110470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854005040 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485e42c700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4854013510 tx=0x7f485400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4854009320 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4858106ba0 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4854013f00 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4854014dc0 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.321+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4858107030 con 0x7f48580f62a0 2023-11-16T01:11:51.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.322+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4854014420 con 0x7f48580f62a0 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.322+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858045ad0 con 0x7f48580f62a0 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.322+0000 7f484effd700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 0x7f4844032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.322+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4854041050 con 0x7f48580f62a0 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.322+0000 7f485dc2b700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 0x7f4844032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.323+0000 7f485dc2b700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 0x7f4844032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f48480096a0 tx=0x7f4848006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:51.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.327+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48540095f0 con 0x7f48580f62a0 2023-11-16T01:11:51.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.530+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4858107ee0 con 0x7f48580f62a0 2023-11-16T01:11:51.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.531+0000 7f484effd700 1 -- 172.21.15.134:0/3523110470 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f485401cb10 con 0x7f48580f62a0 2023-11-16T01:11:51.532 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:51.532 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 msgr2=0x7f4844032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 0x7f4844032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f48480096a0 tx=0x7f4848006b40).stop 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 msgr2=0x7f48581064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4854013510 tx=0x7f485400b040).stop 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 shutdown_connections 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f484402ff60 0x7f4844032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 --2- 172.21.15.134:0/3523110470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48580f62a0 0x7f48581064b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 >> 172.21.15.134:0/3523110470 conn(0x7f48580f1aa0 msgr2=0x7f48580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 shutdown_connections 2023-11-16T01:11:51.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:51.533+0000 7f485f42e700 1 -- 172.21.15.134:0/3523110470 wait complete. 2023-11-16T01:11:51.535 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:52.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:49.960976+0000 mgr.smithi133.xdyttl (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:52.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:51.531750+0000 mon.smithi133 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.134:0/3523110470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:52.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:52.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:54.095 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.094+0000 7f31ab4c0700 1 -- 172.21.15.134:0/3458869249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 msgr2=0x7f31a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:54.095 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.094+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/3458869249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a40f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f3194009fb0 tx=0x7f3194009b30).stop 2023-11-16T01:11:54.095 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.094+0000 7f31ab4c0700 1 -- 172.21.15.134:0/3458869249 shutdown_connections 2023-11-16T01:11:54.095 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.094+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/3458869249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a40f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:54.095 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.094+0000 7f31ab4c0700 1 -- 172.21.15.134:0/3458869249 >> 172.21.15.134:0/3458869249 conn(0x7f31a40f1c70 msgr2=0x7f31a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:54.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 -- 172.21.15.134:0/3458869249 shutdown_connections 2023-11-16T01:11:54.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 -- 172.21.15.134:0/3458869249 wait complete. 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 Processor -- start 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 -- start start 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.095+0000 7f31ab4c0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4106c20 con 0x7f31a40f6510 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31a925c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:54.097 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31a925c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42856/0 (socket says 172.21.15.134:42856) 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31a925c700 1 -- 172.21.15.134:0/1634949622 learned_addr learned my addr 172.21.15.134:0/1634949622 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31a925c700 1 -- 172.21.15.134:0/1634949622 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194005040 con 0x7f31a40f6510 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31a925c700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3194013510 tx=0x7f319400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3194009320 con 0x7f31a40f6510 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a4106e20 con 0x7f31a40f6510 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3194013f00 con 0x7f31a40f6510 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3194014d90 con 0x7f31a40f6510 2023-11-16T01:11:54.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.096+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a41072b0 con 0x7f31a40f6510 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.097+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31940095f0 con 0x7f31a40f6510 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.097+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4045ad0 con 0x7f31a40f6510 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.097+0000 7f3199ffb700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 0x7f3190032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.098+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3194040fa0 con 0x7f31a40f6510 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.098+0000 7f31a8a5b700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 0x7f3190032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:54.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.098+0000 7f31a8a5b700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 0x7f3190032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f31a00096a0 tx=0x7f31a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:54.102 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.102+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f319406d2b0 con 0x7f31a40f6510 2023-11-16T01:11:54.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:51.961256+0000 mgr.smithi133.xdyttl (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:54.307 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.307+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31a40fbf60 con 0x7f31a40f6510 2023-11-16T01:11:54.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.307+0000 7f3199ffb700 1 -- 172.21.15.134:0/1634949622 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3194014420 con 0x7f31a40f6510 2023-11-16T01:11:54.309 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:54.309 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:54.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 msgr2=0x7f3190032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:54.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 0x7f3190032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f31a00096a0 tx=0x7f31a0006b40).stop 2023-11-16T01:11:54.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 msgr2=0x7f31a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:54.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3194013510 tx=0x7f319400b040).stop 2023-11-16T01:11:54.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 shutdown_connections 2023-11-16T01:11:54.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f319002ffb0 0x7f3190032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:54.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 --2- 172.21.15.134:0/1634949622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31a40f6510 0x7f31a4106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:54.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 >> 172.21.15.134:0/1634949622 conn(0x7f31a40f1c70 msgr2=0x7f31a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:54.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 shutdown_connections 2023-11-16T01:11:54.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:54.309+0000 7f31ab4c0700 1 -- 172.21.15.134:0/1634949622 wait complete. 2023-11-16T01:11:54.313 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:55.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:54 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:54.307883+0000 mon.smithi133 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.134:0/1634949622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:55.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:55.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:56.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:53.961526+0000 mgr.smithi133.xdyttl (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:56.965 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 -- 172.21.15.134:0/1027731525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 msgr2=0x7f95980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:56.965 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 --2- 172.21.15.134:0/1027731525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f95980f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9588009fb0 tx=0x7f9588009b30).stop 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 -- 172.21.15.134:0/1027731525 shutdown_connections 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 --2- 172.21.15.134:0/1027731525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f95980f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 -- 172.21.15.134:0/1027731525 >> 172.21.15.134:0/1027731525 conn(0x7f95980f0b70 msgr2=0x7f95980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 -- 172.21.15.134:0/1027731525 shutdown_connections 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.964+0000 7f959e4e4700 1 -- 172.21.15.134:0/1027731525 wait complete. 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f959e4e4700 1 Processor -- start 2023-11-16T01:11:56.966 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f959e4e4700 1 -- start start 2023-11-16T01:11:56.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f959e4e4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:56.967 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f959e4e4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9598106c00 con 0x7f95980f5410 2023-11-16T01:11:56.968 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f9597fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:56.968 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f9597fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42870/0 (socket says 172.21.15.134:42870) 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.965+0000 7f9597fff700 1 -- 172.21.15.134:0/2790850929 learned_addr learned my addr 172.21.15.134:0/2790850929 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f9597fff700 1 -- 172.21.15.134:0/2790850929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9588005040 con 0x7f95980f5410 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f9597fff700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9588013510 tx=0x7f958800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9588009320 con 0x7f95980f5410 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9588013f00 con 0x7f95980f5410 2023-11-16T01:11:56.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598106e00 con 0x7f95980f5410 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9588014d90 con 0x7f95980f5410 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.966+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598107290 con 0x7f95980f5410 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.967+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9588014420 con 0x7f95980f5410 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.967+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9598045ad0 con 0x7f95980f5410 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.967+0000 7f9594ff9700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 0x7f9584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:56.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.967+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95880410c0 con 0x7f95980f5410 2023-11-16T01:11:56.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.967+0000 7f95977fe700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 0x7f9584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:56.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.968+0000 7f95977fe700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 0x7f9584032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f95800096a0 tx=0x7f9580006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:56.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:56.972+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95880095f0 con 0x7f95980f5410 2023-11-16T01:11:57.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.176+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95980fae60 con 0x7f95980f5410 2023-11-16T01:11:57.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.177+0000 7f9594ff9700 1 -- 172.21.15.134:0/2790850929 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f958801db70 con 0x7f95980f5410 2023-11-16T01:11:57.178 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:57.178 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 msgr2=0x7f9584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 0x7f9584032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f95800096a0 tx=0x7f9580006b40).stop 2023-11-16T01:11:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 msgr2=0x7f9598106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9588013510 tx=0x7f958800b040).stop 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 shutdown_connections 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f958402ff60 0x7f9584032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 --2- 172.21.15.134:0/2790850929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f95980f5410 0x7f9598106710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 >> 172.21.15.134:0/2790850929 conn(0x7f95980f0b70 msgr2=0x7f959805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 shutdown_connections 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:57.179+0000 7f959e4e4700 1 -- 172.21.15.134:0/2790850929 wait complete. 2023-11-16T01:11:57.180 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:11:58.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:55.961786+0000 mgr.smithi133.xdyttl (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:11:58.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:57.177561+0000 mon.smithi133 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.134:0/2790850929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:11:58.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:11:58.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:11:59.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 -- 172.21.15.134:0/2747462014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 msgr2=0x7fc0040f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:59.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 --2- 172.21.15.134:0/2747462014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0040f5bd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbff4009fb0 tx=0x7fbff4009b30).stop 2023-11-16T01:11:59.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 -- 172.21.15.134:0/2747462014 shutdown_connections 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 --2- 172.21.15.134:0/2747462014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0040f5bd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 -- 172.21.15.134:0/2747462014 >> 172.21.15.134:0/2747462014 conn(0x7fc0040f0120 msgr2=0x7fc0040f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 -- 172.21.15.134:0/2747462014 shutdown_connections 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.761+0000 7fc003fff700 1 -- 172.21.15.134:0/2747462014 wait complete. 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc003fff700 1 Processor -- start 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc003fff700 1 -- start start 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc003fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc003fff700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc004106990 con 0x7fc0040f57b0 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc002ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc002ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56860/0 (socket says 172.21.15.134:56860) 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.762+0000 7fc002ffd700 1 -- 172.21.15.134:0/1612828238 learned_addr learned my addr 172.21.15.134:0/1612828238 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:11:59.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fc002ffd700 1 -- 172.21.15.134:0/1612828238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4005040 con 0x7fc0040f57b0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fc002ffd700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbff4013510 tx=0x7fbff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff4009320 con 0x7fc0040f57b0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc004106b90 con 0x7fc0040f57b0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbff4013f00 con 0x7fc0040f57b0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbff4014dc0 con 0x7fc0040f57b0 2023-11-16T01:11:59.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.763+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc004107020 con 0x7fc0040f57b0 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.764+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbff401d3d0 con 0x7fc0040f57b0 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.764+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc004045ad0 con 0x7fc0040f57b0 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.764+0000 7fbfeb7fe700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 0x7fbfec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.764+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbff4041300 con 0x7fc0040f57b0 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.764+0000 7fc0027fc700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 0x7fbfec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:11:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.765+0000 7fc0027fc700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 0x7fbfec032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbff80096a0 tx=0x7fbff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:11:59.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.769+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbff401cb10 con 0x7fc0040f57b0 2023-11-16T01:11:59.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.972+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc004107ed0 con 0x7fc0040f57b0 2023-11-16T01:11:59.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.973+0000 7fbfeb7fe700 1 -- 172.21.15.134:0/1612828238 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbff401cb10 con 0x7fc0040f57b0 2023-11-16T01:11:59.974 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:11:59.974 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:11:59.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 msgr2=0x7fbfec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:59.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 0x7fbfec032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbff80096a0 tx=0x7fbff8006b40).stop 2023-11-16T01:11:59.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 msgr2=0x7fc0041064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:11:59.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbff4013510 tx=0x7fbff400b040).stop 2023-11-16T01:11:59.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 shutdown_connections 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbfec02ff60 0x7fbfec032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 --2- 172.21.15.134:0/1612828238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0040f57b0 0x7fc0041064a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 >> 172.21.15.134:0/1612828238 conn(0x7fc0040f0120 msgr2=0x7fc0040f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 shutdown_connections 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:11:59.975+0000 7fc003fff700 1 -- 172.21.15.134:0/1612828238 wait complete. 2023-11-16T01:11:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:00.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:11:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:57.962011+0000 mgr.smithi133.xdyttl (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:01.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:11:59.973603+0000 mon.smithi133 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.134:0/1612828238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:01.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:01.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:02.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:11:59.962346+0000 mgr.smithi133.xdyttl (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2963478556 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 msgr2=0x7f7c980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2963478556 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c980f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f7c8c009fb0 tx=0x7f7c8c009b30).stop 2023-11-16T01:12:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2963478556 shutdown_connections 2023-11-16T01:12:02.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2963478556 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c980f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2963478556 >> 172.21.15.134:0/2963478556 conn(0x7f7c980f1c70 msgr2=0x7f7c980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2963478556 shutdown_connections 2023-11-16T01:12:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.567+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2963478556 wait complete. 2023-11-16T01:12:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9edc0700 1 Processor -- start 2023-11-16T01:12:02.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9edc0700 1 -- start start 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9edc0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9edc0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c98106bc0 con 0x7f7c980f6510 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9cb5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9cb5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56888/0 (socket says 172.21.15.134:56888) 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.568+0000 7f7c9cb5c700 1 -- 172.21.15.134:0/2546369473 learned_addr learned my addr 172.21.15.134:0/2546369473 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c9cb5c700 1 -- 172.21.15.134:0/2546369473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c005040 con 0x7f7c980f6510 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c9cb5c700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7c8c013510 tx=0x7f7c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c009320 con 0x7f7c980f6510 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c98106dc0 con 0x7f7c980f6510 2023-11-16T01:12:02.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c8c013f00 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c014d90 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.569+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c98107250 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.570+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c8c0095f0 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.570+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c98045ad0 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.570+0000 7f7c957fa700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.570+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c8c0410c0 con 0x7f7c980f6510 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.570+0000 7f7c97fff700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:02.572 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.571+0000 7f7c97fff700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7c840096a0 tx=0x7f7c84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:02.575 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.575+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c8c014420 con 0x7f7c980f6510 2023-11-16T01:12:02.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.777+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c98108100 con 0x7f7c980f6510 2023-11-16T01:12:02.778 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.778+0000 7f7c957fa700 1 -- 172.21.15.134:0/2546369473 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c8c010230 con 0x7f7c980f6510 2023-11-16T01:12:02.778 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:02.778 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:02.779 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 msgr2=0x7f7c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7c840096a0 tx=0x7f7c84006b40).stop 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 msgr2=0x7f7c981066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7c8c013510 tx=0x7f7c8c00b040).stop 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 shutdown_connections 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 --2- 172.21.15.134:0/2546369473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7c980f6510 0x7f7c981066d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 >> 172.21.15.134:0/2546369473 conn(0x7f7c980f1c70 msgr2=0x7f7c980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 shutdown_connections 2023-11-16T01:12:02.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:02.779+0000 7f7c9edc0700 1 -- 172.21.15.134:0/2546369473 wait complete. 2023-11-16T01:12:02.781 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:03.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:02.778346+0000 mon.smithi133 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.134:0/2546369473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:04.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:04.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:04.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:01.962622+0000 mgr.smithi133.xdyttl (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 -- 172.21.15.134:0/23258801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 msgr2=0x7fcf200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/23258801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf200f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fcf10009fb0 tx=0x7fcf10009b30).stop 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 -- 172.21.15.134:0/23258801 shutdown_connections 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/23258801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf200f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 -- 172.21.15.134:0/23258801 >> 172.21.15.134:0/23258801 conn(0x7fcf200f0b70 msgr2=0x7fcf200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 -- 172.21.15.134:0/23258801 shutdown_connections 2023-11-16T01:12:05.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 -- 172.21.15.134:0/23258801 wait complete. 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.429+0000 7fcf24d5d700 1 Processor -- start 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf24d5d700 1 -- start start 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf24d5d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf24d5d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20106c20 con 0x7fcf200f5410 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf1e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf1e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56912/0 (socket says 172.21.15.134:56912) 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf1e59c700 1 -- 172.21.15.134:0/3482192340 learned_addr learned my addr 172.21.15.134:0/3482192340 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf1e59c700 1 -- 172.21.15.134:0/3482192340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf10005040 con 0x7fcf200f5410 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.430+0000 7fcf1e59c700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcf10013510 tx=0x7fcf1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.431+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf10009320 con 0x7fcf200f5410 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.431+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf20106e20 con 0x7fcf200f5410 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.431+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf10013f00 con 0x7fcf200f5410 2023-11-16T01:12:05.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.431+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf10014d90 con 0x7fcf200f5410 2023-11-16T01:12:05.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.431+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf201072b0 con 0x7fcf200f5410 2023-11-16T01:12:05.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.432+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf10014420 con 0x7fcf200f5410 2023-11-16T01:12:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.432+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf20045ad0 con 0x7fcf200f5410 2023-11-16T01:12:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.432+0000 7fcf0effd700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 0x7fcf08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.432+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf10041000 con 0x7fcf200f5410 2023-11-16T01:12:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.432+0000 7fcf1dd9b700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 0x7fcf08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.433+0000 7fcf1dd9b700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 0x7fcf08032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:05.438 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.436+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf100095f0 con 0x7fcf200f5410 2023-11-16T01:12:05.646 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.644+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf200fae60 con 0x7fcf200f5410 2023-11-16T01:12:05.646 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.645+0000 7fcf0effd700 1 -- 172.21.15.134:0/3482192340 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcf1001db70 con 0x7fcf200f5410 2023-11-16T01:12:05.646 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:05.647 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.646+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 msgr2=0x7fcf08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.646+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 0x7fcf08032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).stop 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.646+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 msgr2=0x7fcf20106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.646+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcf10013510 tx=0x7fcf1000b040).stop 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 shutdown_connections 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcf0802ff60 0x7fcf08032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:05.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 --2- 172.21.15.134:0/3482192340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf200f5410 0x7fcf20106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:05.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 >> 172.21.15.134:0/3482192340 conn(0x7fcf200f0b70 msgr2=0x7fcf200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:05.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 shutdown_connections 2023-11-16T01:12:05.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:05.647+0000 7fcf24d5d700 1 -- 172.21.15.134:0/3482192340 wait complete. 2023-11-16T01:12:05.649 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:06.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:03.962854+0000 mgr.smithi133.xdyttl (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:06.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:05.646584+0000 mon.smithi133 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.134:0/3482192340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:07.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:07.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:08.276 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.272+0000 7fd33ceec700 1 -- 172.21.15.134:0/3884459460 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 msgr2=0x7fd3380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.272+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3884459460 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd3380f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd328009fb0 tx=0x7fd328009b30).stop 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- 172.21.15.134:0/3884459460 shutdown_connections 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3884459460 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd3380f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- 172.21.15.134:0/3884459460 >> 172.21.15.134:0/3884459460 conn(0x7fd3380f0b70 msgr2=0x7fd3380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- 172.21.15.134:0/3884459460 shutdown_connections 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- 172.21.15.134:0/3884459460 wait complete. 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 Processor -- start 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- start start 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33ceec700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd338060240 con 0x7fd3380f5410 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56932/0 (socket says 172.21.15.134:56932) 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.273+0000 7fd33659c700 1 -- 172.21.15.134:0/3205688336 learned_addr learned my addr 172.21.15.134:0/3205688336 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:08.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd33659c700 1 -- 172.21.15.134:0/3205688336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328005040 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd33659c700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fd328013510 tx=0x7fd32800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd328009320 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd338060440 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd328013f00 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd328014d90 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.274+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd338109ae0 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.275+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3280095f0 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.275+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd338045ad0 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.275+0000 7fd32effd700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 0x7fd324032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.275+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3280410a0 con 0x7fd3380f5410 2023-11-16T01:12:08.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.275+0000 7fd335d9b700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 0x7fd324032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:08.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.276+0000 7fd335d9b700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 0x7fd324032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd3200096a0 tx=0x7fd320006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:08.279 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:05.963145+0000 mgr.smithi133.xdyttl (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:08.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.280+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd32801d7f0 con 0x7fd3380f5410 2023-11-16T01:12:08.488 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.487+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3380fae60 con 0x7fd3380f5410 2023-11-16T01:12:08.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.488+0000 7fd32effd700 1 -- 172.21.15.134:0/3205688336 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd328010230 con 0x7fd3380f5410 2023-11-16T01:12:08.490 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:08.490 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:08.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 msgr2=0x7fd324032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:08.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 0x7fd324032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd3200096a0 tx=0x7fd320006b40).stop 2023-11-16T01:12:08.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 msgr2=0x7fd338109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:08.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fd328013510 tx=0x7fd32800b040).stop 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 shutdown_connections 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd32402ff60 0x7fd324032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 --2- 172.21.15.134:0/3205688336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3380f5410 0x7fd338109430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 >> 172.21.15.134:0/3205688336 conn(0x7fd3380f0b70 msgr2=0x7fd3380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 shutdown_connections 2023-11-16T01:12:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:08.490+0000 7fd33ceec700 1 -- 172.21.15.134:0/3205688336 wait complete. 2023-11-16T01:12:08.494 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:09.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:08 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:08.489477+0000 mon.smithi133 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.134:0/3205688336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:09.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:09.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:07.963491+0000 mgr.smithi133.xdyttl (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 -- 172.21.15.134:0/3631008413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 msgr2=0x7fa1800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3631008413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa1800f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa174009fb0 tx=0x7fa174009b30).stop 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 -- 172.21.15.134:0/3631008413 shutdown_connections 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3631008413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa1800f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 -- 172.21.15.134:0/3631008413 >> 172.21.15.134:0/3631008413 conn(0x7fa1800f0b70 msgr2=0x7fa1800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 -- 172.21.15.134:0/3631008413 shutdown_connections 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.058+0000 7fa186ff1700 1 -- 172.21.15.134:0/3631008413 wait complete. 2023-11-16T01:12:11.060 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa186ff1700 1 Processor -- start 2023-11-16T01:12:11.061 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa186ff1700 1 -- start start 2023-11-16T01:12:11.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa186ff1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:11.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa186ff1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180105b20 con 0x7fa1800f5410 2023-11-16T01:12:11.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa184d8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:11.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa184d8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34916/0 (socket says 172.21.15.134:34916) 2023-11-16T01:12:11.062 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.059+0000 7fa184d8d700 1 -- 172.21.15.134:0/3287637074 learned_addr learned my addr 172.21.15.134:0/3287637074 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa184d8d700 1 -- 172.21.15.134:0/3287637074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174005040 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa184d8d700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa174013510 tx=0x7fa17400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa174009320 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa174013f00 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa180105d20 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1801061b0 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.060+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa174014d90 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1740263a0 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180045ad0 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa17d7fa700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 0x7fa170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1740413c0 con 0x7fa1800f5410 2023-11-16T01:12:11.063 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa17ffff700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 0x7fa170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:11.064 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.061+0000 7fa17ffff700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 0x7fa170032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa16c0096a0 tx=0x7fa16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:11.067 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.066+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa174012510 con 0x7fa1800f5410 2023-11-16T01:12:11.274 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.273+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1800fae60 con 0x7fa1800f5410 2023-11-16T01:12:11.275 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.273+0000 7fa17d7fa700 1 -- 172.21.15.134:0/3287637074 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa174012330 con 0x7fa1800f5410 2023-11-16T01:12:11.275 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:11.275 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:11.276 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 msgr2=0x7fa170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 0x7fa170032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa16c0096a0 tx=0x7fa16c006b40).stop 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 msgr2=0x7fa180105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa174013510 tx=0x7fa17400b040).stop 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 shutdown_connections 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa17002ff60 0x7fa170032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 --2- 172.21.15.134:0/3287637074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1800f5410 0x7fa180105630 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 >> 172.21.15.134:0/3287637074 conn(0x7fa1800f0b70 msgr2=0x7fa1800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 shutdown_connections 2023-11-16T01:12:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:11.275+0000 7fa186ff1700 1 -- 172.21.15.134:0/3287637074 wait complete. 2023-11-16T01:12:11.278 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:12.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:09.963786+0000 mgr.smithi133.xdyttl (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:12.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:11.274818+0000 mon.smithi133 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.134:0/3287637074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:12.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:12.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.889+0000 7f30b9da1700 1 -- 172.21.15.134:0/404424521 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 msgr2=0x7f30b40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.889+0000 7f30b9da1700 1 --2- 172.21.15.134:0/404424521 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b40f8f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f30a8009fb0 tx=0x7f30a8009b30).stop 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.889+0000 7f30b9da1700 1 -- 172.21.15.134:0/404424521 shutdown_connections 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.889+0000 7f30b9da1700 1 --2- 172.21.15.134:0/404424521 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b40f8f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.889+0000 7f30b9da1700 1 -- 172.21.15.134:0/404424521 >> 172.21.15.134:0/404424521 conn(0x7f30b40f09a0 msgr2=0x7f30b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 -- 172.21.15.134:0/404424521 shutdown_connections 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 -- 172.21.15.134:0/404424521 wait complete. 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 Processor -- start 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 -- start start 2023-11-16T01:12:13.892 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:13.893 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.890+0000 7f30b9da1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41069f0 con 0x7f30b40601e0 2023-11-16T01:12:13.893 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b8d9f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:13.893 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b8d9f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34934/0 (socket says 172.21.15.134:34934) 2023-11-16T01:12:13.893 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b8d9f700 1 -- 172.21.15.134:0/3588247171 learned_addr learned my addr 172.21.15.134:0/3588247171 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:13.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b8d9f700 1 -- 172.21.15.134:0/3588247171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a8005040 con 0x7f30b40601e0 2023-11-16T01:12:13.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b8d9f700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f30a8013510 tx=0x7f30a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:13.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a8009320 con 0x7f30b40601e0 2023-11-16T01:12:13.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b4106bf0 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30a8013f00 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a8014d90 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.891+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b4107080 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.892+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30a80095f0 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.892+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b4045ad0 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.892+0000 7f30b17fa700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.892+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30a8041000 con 0x7f30b40601e0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.892+0000 7f30b3fff700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:13.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.893+0000 7f30b3fff700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 0x7f30a4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:13.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:13.897+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30a8014420 con 0x7f30b40601e0 2023-11-16T01:12:14.102 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.100+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30b4107f30 con 0x7f30b40601e0 2023-11-16T01:12:14.102 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.101+0000 7f30b17fa700 1 -- 172.21.15.134:0/3588247171 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30a801d7f0 con 0x7f30b40601e0 2023-11-16T01:12:14.103 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:14.103 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.102+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 msgr2=0x7f30a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.102+0000 7f30b9da1700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 0x7f30a4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).stop 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 msgr2=0x7f30b4106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f30a8013510 tx=0x7f30a800b040).stop 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 shutdown_connections 2023-11-16T01:12:14.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:14.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 --2- 172.21.15.134:0/3588247171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30b40601e0 0x7f30b4106500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:14.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 >> 172.21.15.134:0/3588247171 conn(0x7f30b40f09a0 msgr2=0x7f30b40f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:14.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 shutdown_connections 2023-11-16T01:12:14.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:14.103+0000 7f30b9da1700 1 -- 172.21.15.134:0/3588247171 wait complete. 2023-11-16T01:12:14.105 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:14.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:11.964121+0000 mgr.smithi133.xdyttl (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:15.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:14 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:14.102682+0000 mon.smithi133 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.134:0/3588247171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:15.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:15.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:16.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:13.964475+0000 mgr.smithi133.xdyttl (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:16.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 -- 172.21.15.134:0/3159029944 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 msgr2=0x7f81e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:16.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 --2- 172.21.15.134:0/3159029944 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e40f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f81d4009fb0 tx=0x7f81d4009b30).stop 2023-11-16T01:12:16.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 -- 172.21.15.134:0/3159029944 shutdown_connections 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 --2- 172.21.15.134:0/3159029944 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e40f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 -- 172.21.15.134:0/3159029944 >> 172.21.15.134:0/3159029944 conn(0x7f81e40f1c70 msgr2=0x7f81e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 -- 172.21.15.134:0/3159029944 shutdown_connections 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.763+0000 7f81ea371700 1 -- 172.21.15.134:0/3159029944 wait complete. 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81ea371700 1 Processor -- start 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81ea371700 1 -- start start 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81ea371700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81ea371700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e4106bc0 con 0x7f81e40f6510 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81e3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81e3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34942/0 (socket says 172.21.15.134:34942) 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.764+0000 7f81e3fff700 1 -- 172.21.15.134:0/3371962985 learned_addr learned my addr 172.21.15.134:0/3371962985 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81e3fff700 1 -- 172.21.15.134:0/3371962985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4005040 con 0x7f81e40f6510 2023-11-16T01:12:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81e3fff700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f81d4013510 tx=0x7f81d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d4009320 con 0x7f81e40f6510 2023-11-16T01:12:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81d4013f00 con 0x7f81e40f6510 2023-11-16T01:12:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e4106dc0 con 0x7f81e40f6510 2023-11-16T01:12:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d4014d90 con 0x7f81e40f6510 2023-11-16T01:12:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.765+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e4107250 con 0x7f81e40f6510 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81d40263a0 con 0x7f81e40f6510 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81e0ff9700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 0x7f81cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81d4041100 con 0x7f81e40f6510 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e4045ad0 con 0x7f81e40f6510 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81e37fe700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 0x7f81cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:16.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.766+0000 7f81e37fe700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 0x7f81cc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f81d80096a0 tx=0x7f81d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:16.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.771+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81d4012920 con 0x7f81e40f6510 2023-11-16T01:12:16.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.974+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81e4107fa0 con 0x7f81e40f6510 2023-11-16T01:12:16.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.975+0000 7f81e0ff9700 1 -- 172.21.15.134:0/3371962985 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f81d402b020 con 0x7f81e40f6510 2023-11-16T01:12:16.978 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:16.978 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:16.979 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 msgr2=0x7f81cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:16.979 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 0x7f81cc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f81d80096a0 tx=0x7f81d8006b40).stop 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 msgr2=0x7f81e41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f81d4013510 tx=0x7f81d400b040).stop 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 shutdown_connections 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81cc02ff60 0x7f81cc032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 --2- 172.21.15.134:0/3371962985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81e40f6510 0x7f81e41066d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 >> 172.21.15.134:0/3371962985 conn(0x7f81e40f1c70 msgr2=0x7f81e40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 shutdown_connections 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:16.976+0000 7f81ea371700 1 -- 172.21.15.134:0/3371962985 wait complete. 2023-11-16T01:12:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:15.964810+0000 mgr.smithi133.xdyttl (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:16.976172+0000 mon.smithi133 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.134:0/3371962985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:18.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:18.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 -- 172.21.15.134:0/3939880698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 msgr2=0x7f9d880f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/3939880698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d880f4f20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f9d7c004770 tx=0x7f9d7c00ac40).stop 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 -- 172.21.15.134:0/3939880698 shutdown_connections 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/3939880698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d880f4f20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 -- 172.21.15.134:0/3939880698 >> 172.21.15.134:0/3939880698 conn(0x7f9d880f0260 msgr2=0x7f9d880f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.649+0000 7f9d8cd03700 1 -- 172.21.15.134:0/3939880698 shutdown_connections 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.650+0000 7f9d8cd03700 1 -- 172.21.15.134:0/3939880698 wait complete. 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.650+0000 7f9d8cd03700 1 Processor -- start 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.650+0000 7f9d8cd03700 1 -- start start 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.650+0000 7f9d8cd03700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.650+0000 7f9d8cd03700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88106c10 con 0x7f9d880f4b00 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46962/0 (socket says 172.21.15.134:46962) 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8659c700 1 -- 172.21.15.134:0/2051848562 learned_addr learned my addr 172.21.15.134:0/2051848562 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8659c700 1 -- 172.21.15.134:0/2051848562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c005040 con 0x7f9d880f4b00 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8659c700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f9d7c004740 tx=0x7f9d7c00ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d7c019070 con 0x7f9d880f4b00 2023-11-16T01:12:19.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88106e10 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d7c007e70 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d7c0082d0 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.651+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d881072a0 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.652+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d7c008560 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.652+0000 7f9d76ffd700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.652+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d7c041080 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.652+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d88045ad0 con 0x7f9d880f4b00 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.652+0000 7f9d85d9b700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.653+0000 7f9d85d9b700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 0x7f9d70032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9d780096a0 tx=0x7f9d78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:19.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.657+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d7c0133c0 con 0x7f9d880f4b00 2023-11-16T01:12:19.863 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.861+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d88107fc0 con 0x7f9d880f4b00 2023-11-16T01:12:19.863 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.862+0000 7f9d76ffd700 1 -- 172.21.15.134:0/2051848562 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d7c01b2d0 con 0x7f9d880f4b00 2023-11-16T01:12:19.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:19.864 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 msgr2=0x7f9d70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 0x7f9d70032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9d780096a0 tx=0x7f9d78006b40).stop 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 msgr2=0x7f9d88106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f9d7c004740 tx=0x7f9d7c00ac40).stop 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 shutdown_connections 2023-11-16T01:12:19.865 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 --2- 172.21.15.134:0/2051848562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d880f4b00 0x7f9d88106720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 >> 172.21.15.134:0/2051848562 conn(0x7f9d880f0260 msgr2=0x7f9d880f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 shutdown_connections 2023-11-16T01:12:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:19.863+0000 7f9d8cd03700 1 -- 172.21.15.134:0/2051848562 wait complete. 2023-11-16T01:12:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:20.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:17.965162+0000 mgr.smithi133.xdyttl (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:20.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:19.863341+0000 mon.smithi133 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.134:0/2051848562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:21.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:21.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:22.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:19.965444+0000 mgr.smithi133.xdyttl (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:22.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.424+0000 7f3df6edd700 1 -- 172.21.15.134:0/3401839896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 msgr2=0x7f3df00f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.424+0000 7f3df6edd700 1 --2- 172.21.15.134:0/3401839896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df00f8fe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f3de0009fb0 tx=0x7f3de0009b30).stop 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- 172.21.15.134:0/3401839896 shutdown_connections 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 --2- 172.21.15.134:0/3401839896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df00f8fe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- 172.21.15.134:0/3401839896 >> 172.21.15.134:0/3401839896 conn(0x7f3df00f09a0 msgr2=0x7f3df00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- 172.21.15.134:0/3401839896 shutdown_connections 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- 172.21.15.134:0/3401839896 wait complete. 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 Processor -- start 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- start start 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df6edd700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df01069f0 con 0x7f3df0060120 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df5edb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df5edb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46978/0 (socket says 172.21.15.134:46978) 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.425+0000 7f3df5edb700 1 -- 172.21.15.134:0/4106245022 learned_addr learned my addr 172.21.15.134:0/4106245022 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3df5edb700 1 -- 172.21.15.134:0/4106245022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0005040 con 0x7f3df0060120 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3df5edb700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f3de0013510 tx=0x7f3de000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de0009320 con 0x7f3df0060120 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3de0013f00 con 0x7f3df0060120 2023-11-16T01:12:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df0106bf0 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de0014d90 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.426+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df0107080 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.427+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3de00095f0 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.427+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0045ad0 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.427+0000 7f3de67fc700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 0x7f3ddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.427+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3de00410c0 con 0x7f3df0060120 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.427+0000 7f3df56da700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 0x7f3ddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:22.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.428+0000 7f3df56da700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 0x7f3ddc032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3dec0096a0 tx=0x7f3dec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:22.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.432+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3de0014420 con 0x7f3df0060120 2023-11-16T01:12:22.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.639+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3df0107f30 con 0x7f3df0060120 2023-11-16T01:12:22.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.639+0000 7f3de67fc700 1 -- 172.21.15.134:0/4106245022 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3de001d7f0 con 0x7f3df0060120 2023-11-16T01:12:22.641 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:22.641 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 msgr2=0x7f3ddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 0x7f3ddc032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3dec0096a0 tx=0x7f3dec006b40).stop 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 msgr2=0x7f3df0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f3de0013510 tx=0x7f3de000b040).stop 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 shutdown_connections 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3ddc02ff60 0x7f3ddc032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 --2- 172.21.15.134:0/4106245022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df0060120 0x7f3df0106500 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 >> 172.21.15.134:0/4106245022 conn(0x7f3df00f09a0 msgr2=0x7f3df00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 shutdown_connections 2023-11-16T01:12:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:22.641+0000 7f3df6edd700 1 -- 172.21.15.134:0/4106245022 wait complete. 2023-11-16T01:12:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:23.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:22.640989+0000 mon.smithi133 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.134:0/4106245022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:24.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:24.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:24.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:21.965795+0000 mgr.smithi133.xdyttl (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:25.267 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/3209421641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 msgr2=0x7f9dbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:30.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/3209421641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f9dac009fb0 tx=0x7f9dac009b30).stop 2023-11-16T01:12:30.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/3209421641 shutdown_connections 2023-11-16T01:12:30.080 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/3209421641 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/3209421641 >> 172.21.15.134:0/3209421641 conn(0x7f9dbc0f0b70 msgr2=0x7f9dbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/3209421641 shutdown_connections 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.264+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/3209421641 wait complete. 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dc1ef1700 1 Processor -- start 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dc1ef1700 1 -- start start 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dc1ef1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dc1ef1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc105b20 con 0x7f9dbc0f5410 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dbb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dbb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46992/0 (socket says 172.21.15.134:46992) 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.265+0000 7f9dbb7fe700 1 -- 172.21.15.134:0/2233927166 learned_addr learned my addr 172.21.15.134:0/2233927166 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9dbb7fe700 1 -- 172.21.15.134:0/2233927166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac005040 con 0x7f9dbc0f5410 2023-11-16T01:12:30.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9dbb7fe700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9dac013510 tx=0x7f9dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:30.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dac009320 con 0x7f9dbc0f5410 2023-11-16T01:12:30.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc105d20 con 0x7f9dbc0f5410 2023-11-16T01:12:30.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dac013f00 con 0x7f9dbc0f5410 2023-11-16T01:12:30.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dac014d90 con 0x7f9dbc0f5410 2023-11-16T01:12:30.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.266+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc1061b0 con 0x7f9dbc0f5410 2023-11-16T01:12:30.083 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:23.966066+0000 mgr.smithi133.xdyttl (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:30.083 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:25.478487+0000 mon.smithi133 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.134:0/2233927166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:30.083 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:25.966418+0000 mgr.smithi133.xdyttl (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.267+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dac0263a0 con 0x7f9dbc0f5410 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.267+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc045ad0 con 0x7f9dbc0f5410 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.267+0000 7f9da3fff700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 0x7f9da4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.267+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dac040fb0 con 0x7f9dbc0f5410 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.267+0000 7f9dbaffd700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 0x7f9da4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.268+0000 7f9dbaffd700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 0x7f9da4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9db00096a0 tx=0x7f9db0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.272+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dac0095f0 con 0x7f9dbc0f5410 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.476+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dbc0fae60 con 0x7f9dbc0f5410 2023-11-16T01:12:30.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.478+0000 7f9da3fff700 1 -- 172.21.15.134:0/2233927166 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9dac01db70 con 0x7f9dbc0f5410 2023-11-16T01:12:30.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:30.085 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 msgr2=0x7f9da4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 0x7f9da4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9db00096a0 tx=0x7f9db0006b40).stop 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 msgr2=0x7f9dbc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9dac013510 tx=0x7f9dac00b040).stop 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 shutdown_connections 2023-11-16T01:12:30.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da402ff60 0x7f9da4032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:30.087 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 --2- 172.21.15.134:0/2233927166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dbc0f5410 0x7f9dbc105630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:30.087 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 >> 172.21.15.134:0/2233927166 conn(0x7f9dbc0f0b70 msgr2=0x7f9dbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:30.087 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 shutdown_connections 2023-11-16T01:12:30.087 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:25.479+0000 7f9dc1ef1700 1 -- 172.21.15.134:0/2233927166 wait complete. 2023-11-16T01:12:30.141 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:27.966726+0000 mgr.smithi133.xdyttl (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:31.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:31.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:30.073917+0000 mon.smithi133 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:12:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:30.075277+0000 mon.smithi133 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:12:32.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:29.967023+0000 mgr.smithi133.xdyttl (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 -- 172.21.15.134:0/3019322041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 msgr2=0x7f6fe40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/3019322041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe40f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f6fd8009fb0 tx=0x7f6fd8009b30).stop 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 -- 172.21.15.134:0/3019322041 shutdown_connections 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/3019322041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe40f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 -- 172.21.15.134:0/3019322041 >> 172.21.15.134:0/3019322041 conn(0x7f6fe40f09a0 msgr2=0x7f6fe40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 -- 172.21.15.134:0/3019322041 shutdown_connections 2023-11-16T01:12:32.339 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.338+0000 7f6fe9e79700 1 -- 172.21.15.134:0/3019322041 wait complete. 2023-11-16T01:12:32.340 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe9e79700 1 Processor -- start 2023-11-16T01:12:32.340 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe9e79700 1 -- start start 2023-11-16T01:12:32.340 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe9e79700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:32.340 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe9e79700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4060240 con 0x7f6fe40f51a0 2023-11-16T01:12:32.340 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe8e77700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe8e77700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54492/0 (socket says 172.21.15.134:54492) 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.339+0000 7f6fe8e77700 1 -- 172.21.15.134:0/571854488 learned_addr learned my addr 172.21.15.134:0/571854488 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe8e77700 1 -- 172.21.15.134:0/571854488 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8005040 con 0x7f6fe40f51a0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe8e77700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6fd8013510 tx=0x7f6fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8009320 con 0x7f6fe40f51a0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4060440 con 0x7f6fe40f51a0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fd8013f00 con 0x7f6fe40f51a0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8014dc0 con 0x7f6fe40f51a0 2023-11-16T01:12:32.341 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.340+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe4109860 con 0x7f6fe40f51a0 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.341+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fd80095f0 con 0x7f6fe40f51a0 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.341+0000 7f6fe17fa700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 0x7f6fd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.341+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4045ad0 con 0x7f6fe40f51a0 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.341+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fd8041000 con 0x7f6fe40f51a0 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.341+0000 7f6fe3fff700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 0x7f6fd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:32.342 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.342+0000 7f6fe3fff700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 0x7f6fd4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6fd00096a0 tx=0x7f6fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:32.346 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.346+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fd801c790 con 0x7f6fe40f51a0 2023-11-16T01:12:32.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.552+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fe410a080 con 0x7f6fe40f51a0 2023-11-16T01:12:32.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.553+0000 7f6fe17fa700 1 -- 172.21.15.134:0/571854488 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6fd802b020 con 0x7f6fe40f51a0 2023-11-16T01:12:32.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:32.553 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.554+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 msgr2=0x7f6fd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.554+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 0x7f6fd4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6fd00096a0 tx=0x7f6fd0006b40).stop 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.554+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 msgr2=0x7f6fe41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.554+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6fd8013510 tx=0x7f6fd800b040).stop 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 shutdown_connections 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fd402ff60 0x7f6fd4032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:32.555 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 --2- 172.21.15.134:0/571854488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe40f51a0 0x7f6fe41091b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:32.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 >> 172.21.15.134:0/571854488 conn(0x7f6fe40f09a0 msgr2=0x7f6fe40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:32.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 shutdown_connections 2023-11-16T01:12:32.556 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:32.555+0000 7f6fe9e79700 1 -- 172.21.15.134:0/571854488 wait complete. 2023-11-16T01:12:32.556 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:33.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:32.553459+0000 mon.smithi133 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.134:0/571854488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:33.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:33.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:34.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:31.967268+0000 mgr.smithi133.xdyttl (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:35.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.144+0000 7f7b0f59e700 1 -- 172.21.15.134:0/429571838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 msgr2=0x7f7b100f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.144+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/429571838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b100f5560 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7b04009fb0 tx=0x7f7b04009b30).stop 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- 172.21.15.134:0/429571838 shutdown_connections 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/429571838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b100f5560 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- 172.21.15.134:0/429571838 >> 172.21.15.134:0/429571838 conn(0x7f7b100f08c0 msgr2=0x7f7b100f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- 172.21.15.134:0/429571838 shutdown_connections 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- 172.21.15.134:0/429571838 wait complete. 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 Processor -- start 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- start start 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:35.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0f59e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10106990 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54504/0 (socket says 172.21.15.134:54504) 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.145+0000 7f7b0e59c700 1 -- 172.21.15.134:0/2748512801 learned_addr learned my addr 172.21.15.134:0/2748512801 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7b0e59c700 1 -- 172.21.15.134:0/2748512801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04005040 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7b0e59c700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7b04013510 tx=0x7f7b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b04009320 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b04013f00 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b10106b90 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b04014d90 con 0x7f7b100f5140 2023-11-16T01:12:35.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.146+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10107020 con 0x7f7b100f5140 2023-11-16T01:12:35.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.147+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b10045ad0 con 0x7f7b100f5140 2023-11-16T01:12:35.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.147+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b040095f0 con 0x7f7b100f5140 2023-11-16T01:12:35.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.147+0000 7f7afeffd700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 0x7f7af8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.147+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b04041000 con 0x7f7b100f5140 2023-11-16T01:12:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.147+0000 7f7b0dd9b700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 0x7f7af8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.148+0000 7f7b0dd9b700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 0x7f7af8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f7b000096a0 tx=0x7f7b00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:35.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.152+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b0401d7f0 con 0x7f7b100f5140 2023-11-16T01:12:35.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.356+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b10107ed0 con 0x7f7b100f5140 2023-11-16T01:12:35.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.357+0000 7f7afeffd700 1 -- 172.21.15.134:0/2748512801 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7b04010230 con 0x7f7b100f5140 2023-11-16T01:12:35.358 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:35.358 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:35.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 msgr2=0x7f7af8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 0x7f7af8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f7b000096a0 tx=0x7f7b00006b40).stop 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 msgr2=0x7f7b101064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7b04013510 tx=0x7f7b0400b040).stop 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 shutdown_connections 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7af802ff60 0x7f7af8032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 --2- 172.21.15.134:0/2748512801 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b100f5140 0x7f7b101064a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 >> 172.21.15.134:0/2748512801 conn(0x7f7b100f08c0 msgr2=0x7f7b1005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 shutdown_connections 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:35.359+0000 7f7b0f59e700 1 -- 172.21.15.134:0/2748512801 wait complete. 2023-11-16T01:12:35.360 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:36.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:33.967610+0000 mgr.smithi133.xdyttl (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:36.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:35.357769+0000 mon.smithi133 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.134:0/2748512801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:36.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:36.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:38.009 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- 172.21.15.134:0/892721420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 msgr2=0x7f0cf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/892721420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f0ce0009fb0 tx=0x7f0ce0009b30).stop 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- 172.21.15.134:0/892721420 shutdown_connections 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/892721420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- 172.21.15.134:0/892721420 >> 172.21.15.134:0/892721420 conn(0x7f0cf00f0b70 msgr2=0x7f0cf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- 172.21.15.134:0/892721420 shutdown_connections 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- 172.21.15.134:0/892721420 wait complete. 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 Processor -- start 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- start start 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.008+0000 7f0cf74d7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00fb820 con 0x7f0cf00f5410 2023-11-16T01:12:38.010 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf5273700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf5273700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54524/0 (socket says 172.21.15.134:54524) 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf5273700 1 -- 172.21.15.134:0/1981465156 learned_addr learned my addr 172.21.15.134:0/1981465156 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf5273700 1 -- 172.21.15.134:0/1981465156 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce0005040 con 0x7f0cf00f5410 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf5273700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0ce0013510 tx=0x7f0ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce0009320 con 0x7f0cf00f5410 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf00fb9c0 con 0x7f0cf00f5410 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ce0013f00 con 0x7f0cf00f5410 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce0014d90 con 0x7f0cf00f5410 2023-11-16T01:12:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.009+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00fbe50 con 0x7f0cf00f5410 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.010+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ce00095f0 con 0x7f0cf00f5410 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.010+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf0059100 con 0x7f0cf00f5410 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.010+0000 7f0ce5ffb700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.010+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ce00410c0 con 0x7f0cf00f5410 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.011+0000 7f0cf4a72700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.011+0000 7f0cf4a72700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 0x7f0cdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f0cec0096a0 tx=0x7f0cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.015+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ce001d7f0 con 0x7f0cf00f5410 2023-11-16T01:12:38.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.220+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cf00fae60 con 0x7f0cf00f5410 2023-11-16T01:12:38.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.221+0000 7f0ce5ffb700 1 -- 172.21.15.134:0/1981465156 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ce0010230 con 0x7f0cf00f5410 2023-11-16T01:12:38.222 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:38.222 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:38.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 msgr2=0x7f0cdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:38.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 0x7f0cdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f0cec0096a0 tx=0x7f0cec006b40).stop 2023-11-16T01:12:38.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 msgr2=0x7f0cf00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:38.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0ce0013510 tx=0x7f0ce000b040).stop 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 shutdown_connections 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0cdc02ff60 0x7f0cdc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 --2- 172.21.15.134:0/1981465156 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0cf00f5410 0x7f0cf00fd3a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 >> 172.21.15.134:0/1981465156 conn(0x7f0cf00f0b70 msgr2=0x7f0cf00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 shutdown_connections 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:38.223+0000 7f0cf74d7700 1 -- 172.21.15.134:0/1981465156 wait complete. 2023-11-16T01:12:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:38.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:35.967894+0000 mgr.smithi133.xdyttl (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:39.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:38.221485+0000 mon.smithi133 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.134:0/1981465156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:39.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:39.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:37.968241+0000 mgr.smithi133.xdyttl (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- 172.21.15.134:0/677699832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 msgr2=0x7f272c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 --2- 172.21.15.134:0/677699832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c0f55c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f2720009fb0 tx=0x7f2720009b30).stop 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- 172.21.15.134:0/677699832 shutdown_connections 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 --2- 172.21.15.134:0/677699832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c0f55c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- 172.21.15.134:0/677699832 >> 172.21.15.134:0/677699832 conn(0x7f272c0f09a0 msgr2=0x7f272c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- 172.21.15.134:0/677699832 shutdown_connections 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- 172.21.15.134:0/677699832 wait complete. 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 Processor -- start 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.876+0000 7f2731c0f700 1 -- start start 2023-11-16T01:12:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2731c0f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2731c0f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c060220 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2730c0d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2730c0d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37838/0 (socket says 172.21.15.134:37838) 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2730c0d700 1 -- 172.21.15.134:0/331302116 learned_addr learned my addr 172.21.15.134:0/331302116 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2730c0d700 1 -- 172.21.15.134:0/331302116 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720005040 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2730c0d700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2720013510 tx=0x7f272000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2720009320 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2720013f00 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c060420 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2720014dc0 con 0x7f272c0f51a0 2023-11-16T01:12:40.879 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.877+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c109880 con 0x7f272c0f51a0 2023-11-16T01:12:40.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.878+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2720014420 con 0x7f272c0f51a0 2023-11-16T01:12:40.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.878+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c059100 con 0x7f272c0f51a0 2023-11-16T01:12:40.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.879+0000 7f27297fa700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 0x7f271c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:40.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.879+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2720041050 con 0x7f272c0f51a0 2023-11-16T01:12:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.879+0000 7f272bfff700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 0x7f271c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.879+0000 7f272bfff700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 0x7f271c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f27180096a0 tx=0x7f2718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:40.883+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27200095f0 con 0x7f272c0f51a0 2023-11-16T01:12:41.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.086+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f272c045ad0 con 0x7f272c0f51a0 2023-11-16T01:12:41.087 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.086+0000 7f27297fa700 1 -- 172.21.15.134:0/331302116 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f272001cb10 con 0x7f272c0f51a0 2023-11-16T01:12:41.088 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:41.088 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:41.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 msgr2=0x7f271c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 0x7f271c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f27180096a0 tx=0x7f2718006b40).stop 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 msgr2=0x7f272c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2720013510 tx=0x7f272000b040).stop 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 shutdown_connections 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f271c02ff60 0x7f271c032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 --2- 172.21.15.134:0/331302116 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f272c0f51a0 0x7f272c1091d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 >> 172.21.15.134:0/331302116 conn(0x7f272c0f09a0 msgr2=0x7f272c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 shutdown_connections 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:41.088+0000 7f2731c0f700 1 -- 172.21.15.134:0/331302116 wait complete. 2023-11-16T01:12:41.090 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:42.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:39.968514+0000 mgr.smithi133.xdyttl (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:42.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:41.087016+0000 mon.smithi133 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.134:0/331302116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:42.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:42.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4283048029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 msgr2=0x7fbc200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4283048029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc200f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fbc14009fb0 tx=0x7fbc14009b30).stop 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4283048029 shutdown_connections 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4283048029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc200f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4283048029 >> 172.21.15.134:0/4283048029 conn(0x7fbc200f0b70 msgr2=0x7fbc200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4283048029 shutdown_connections 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.689+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4283048029 wait complete. 2023-11-16T01:12:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc25f7e700 1 Processor -- start 2023-11-16T01:12:43.692 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc25f7e700 1 -- start start 2023-11-16T01:12:43.692 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc25f7e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:43.692 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc25f7e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20060240 con 0x7fbc200f5410 2023-11-16T01:12:43.692 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc1f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:43.692 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc1f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37850/0 (socket says 172.21.15.134:37850) 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.690+0000 7fbc1f7fe700 1 -- 172.21.15.134:0/4046942478 learned_addr learned my addr 172.21.15.134:0/4046942478 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc1f7fe700 1 -- 172.21.15.134:0/4046942478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc14005040 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc1f7fe700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbc14013510 tx=0x7fbc1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc14009320 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc20060440 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc14013f00 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc14014d90 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.691+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc20109a80 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.692+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc140095f0 con 0x7fbc200f5410 2023-11-16T01:12:43.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.692+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc20045ad0 con 0x7fbc200f5410 2023-11-16T01:12:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.692+0000 7fbc07fff700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 0x7fbc08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.692+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc14040fe0 con 0x7fbc200f5410 2023-11-16T01:12:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.692+0000 7fbc1effd700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 0x7fbc08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.693+0000 7fbc1effd700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 0x7fbc08032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbc100096a0 tx=0x7fbc10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:43.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.697+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc1401d7f0 con 0x7fbc200f5410 2023-11-16T01:12:43.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.901+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc2010a2a0 con 0x7fbc200f5410 2023-11-16T01:12:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.903+0000 7fbc07fff700 1 -- 172.21.15.134:0/4046942478 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc14010230 con 0x7fbc200f5410 2023-11-16T01:12:43.904 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:43.904 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 msgr2=0x7fbc08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 0x7fbc08032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbc100096a0 tx=0x7fbc10006b40).stop 2023-11-16T01:12:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 msgr2=0x7fbc201093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:43.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbc14013510 tx=0x7fbc1400b040).stop 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 shutdown_connections 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbc0802ff60 0x7fbc08032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 --2- 172.21.15.134:0/4046942478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbc200f5410 0x7fbc201093d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 >> 172.21.15.134:0/4046942478 conn(0x7fbc200f0b70 msgr2=0x7fbc200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 shutdown_connections 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:43.905+0000 7fbc25f7e700 1 -- 172.21.15.134:0/4046942478 wait complete. 2023-11-16T01:12:43.906 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:41.968760+0000 mgr.smithi133.xdyttl (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:45.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:44 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:43.903811+0000 mon.smithi133 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.134:0/4046942478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:45.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:45.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:43.969098+0000 mgr.smithi133.xdyttl (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:46.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.565+0000 7fb3c61b4700 1 -- 172.21.15.134:0/3531578693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 msgr2=0x7fb3c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:46.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.565+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/3531578693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c00f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb3bc009fb0 tx=0x7fb3bc009b30).stop 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.565+0000 7fb3c61b4700 1 -- 172.21.15.134:0/3531578693 shutdown_connections 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.565+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/3531578693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c00f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.565+0000 7fb3c61b4700 1 -- 172.21.15.134:0/3531578693 >> 172.21.15.134:0/3531578693 conn(0x7fb3c00f09a0 msgr2=0x7fb3c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 -- 172.21.15.134:0/3531578693 shutdown_connections 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 -- 172.21.15.134:0/3531578693 wait complete. 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 Processor -- start 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 -- start start 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c61b4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c0108bb0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c51b2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c51b2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37866/0 (socket says 172.21.15.134:37866) 2023-11-16T01:12:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.566+0000 7fb3c51b2700 1 -- 172.21.15.134:0/2344395414 learned_addr learned my addr 172.21.15.134:0/2344395414 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3c51b2700 1 -- 172.21.15.134:0/2344395414 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc005040 con 0x7fb3c00f51a0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3c51b2700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb3bc013510 tx=0x7fb3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3bc009320 con 0x7fb3c00f51a0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3bc013f00 con 0x7fb3c00f51a0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c0108db0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3bc014dc0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.567+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c0109240 con 0x7fb3c00f51a0 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.568+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3bc01d3d0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.568+0000 7fb3b5ffb700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 0x7fb3ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.568+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c0045ad0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.568+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3bc0413e0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.568+0000 7fb3c49b1700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 0x7fb3ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:46.571 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.569+0000 7fb3c49b1700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 0x7fb3ac032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:46.574 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.573+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3bc026490 con 0x7fb3c00f51a0 2023-11-16T01:12:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.778+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3c010a0f0 con 0x7fb3c00f51a0 2023-11-16T01:12:46.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.779+0000 7fb3b5ffb700 1 -- 172.21.15.134:0/2344395414 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb3bc026490 con 0x7fb3c00f51a0 2023-11-16T01:12:46.780 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:46.780 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 msgr2=0x7fb3ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 0x7fb3ac032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).stop 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 msgr2=0x7fb3c01086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb3bc013510 tx=0x7fb3bc00b040).stop 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 shutdown_connections 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb3ac02ff60 0x7fb3ac032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 --2- 172.21.15.134:0/2344395414 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3c00f51a0 0x7fb3c01086c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 >> 172.21.15.134:0/2344395414 conn(0x7fb3c00f09a0 msgr2=0x7fb3c005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 shutdown_connections 2023-11-16T01:12:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:46.780+0000 7fb3c61b4700 1 -- 172.21.15.134:0/2344395414 wait complete. 2023-11-16T01:12:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:45.969428+0000 mgr.smithi133.xdyttl (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:48.120 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:46.780145+0000 mon.smithi133 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.134:0/2344395414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:48.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:48.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 -- 172.21.15.134:0/2113777558 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 msgr2=0x7f16500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 --2- 172.21.15.134:0/2113777558 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f16500f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f1640009fb0 tx=0x7f1640009b30).stop 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 -- 172.21.15.134:0/2113777558 shutdown_connections 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 --2- 172.21.15.134:0/2113777558 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f16500f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 -- 172.21.15.134:0/2113777558 >> 172.21.15.134:0/2113777558 conn(0x7f16500f0b70 msgr2=0x7f16500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 -- 172.21.15.134:0/2113777558 shutdown_connections 2023-11-16T01:12:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.445+0000 7f1657bec700 1 -- 172.21.15.134:0/2113777558 wait complete. 2023-11-16T01:12:49.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1657bec700 1 Processor -- start 2023-11-16T01:12:49.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1657bec700 1 -- start start 2023-11-16T01:12:49.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1657bec700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:49.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1657bec700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650105b20 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1655988700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1655988700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33568/0 (socket says 172.21.15.134:33568) 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.446+0000 7f1655988700 1 -- 172.21.15.134:0/1061725735 learned_addr learned my addr 172.21.15.134:0/1061725735 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f1655988700 1 -- 172.21.15.134:0/1061725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640005040 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f1655988700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f1640013510 tx=0x7f164000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640009320 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1640013f00 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1650105d20 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640014d90 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.447+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16501061b0 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.448+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1640014420 con 0x7f16500f5410 2023-11-16T01:12:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.448+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1650045ad0 con 0x7f16500f5410 2023-11-16T01:12:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.448+0000 7f16467fc700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.448+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1640041000 con 0x7f16500f5410 2023-11-16T01:12:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.448+0000 7f1655187700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.449+0000 7f1655187700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 0x7f163c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:49.454 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.453+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16400095f0 con 0x7f16500f5410 2023-11-16T01:12:49.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.655+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16500fae60 con 0x7f16500f5410 2023-11-16T01:12:49.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.656+0000 7f16467fc700 1 -- 172.21.15.134:0/1061725735 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f164001db70 con 0x7f16500f5410 2023-11-16T01:12:49.657 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:49.657 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 msgr2=0x7f163c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 0x7f163c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).stop 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 msgr2=0x7f1650105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f1640013510 tx=0x7f164000b040).stop 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 shutdown_connections 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 --2- 172.21.15.134:0/1061725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16500f5410 0x7f1650105630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 >> 172.21.15.134:0/1061725735 conn(0x7f16500f0b70 msgr2=0x7f16500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 shutdown_connections 2023-11-16T01:12:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:49.657+0000 7f1657bec700 1 -- 172.21.15.134:0/1061725735 wait complete. 2023-11-16T01:12:49.660 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:50.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:47.969773+0000 mgr.smithi133.xdyttl (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:51.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:49.657193+0000 mon.smithi133 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.134:0/1061725735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:51.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:51.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:52.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:49.970144+0000 mgr.smithi133.xdyttl (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:52.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.217+0000 7f7aeafa3700 1 -- 172.21.15.134:0/280421294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 msgr2=0x7f7ae40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:52.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.217+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/280421294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae40f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f7ae0009fb0 tx=0x7f7ae0009b30).stop 2023-11-16T01:12:52.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.217+0000 7f7aeafa3700 1 -- 172.21.15.134:0/280421294 shutdown_connections 2023-11-16T01:12:52.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.217+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/280421294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae40f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:52.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.217+0000 7f7aeafa3700 1 -- 172.21.15.134:0/280421294 >> 172.21.15.134:0/280421294 conn(0x7f7ae40f0b70 msgr2=0x7f7ae40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:52.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 -- 172.21.15.134:0/280421294 shutdown_connections 2023-11-16T01:12:52.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 -- 172.21.15.134:0/280421294 wait complete. 2023-11-16T01:12:52.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 Processor -- start 2023-11-16T01:12:52.224 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 -- start start 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.218+0000 7f7aeafa3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4060220 con 0x7f7ae40f5410 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ae8d3f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ae8d3f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33584/0 (socket says 172.21.15.134:33584) 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ae8d3f700 1 -- 172.21.15.134:0/3012426158 learned_addr learned my addr 172.21.15.134:0/3012426158 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:52.225 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ae8d3f700 1 -- 172.21.15.134:0/3012426158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae0005040 con 0x7f7ae40f5410 2023-11-16T01:12:52.226 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ae8d3f700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7ae0014000 tx=0x7f7ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:52.226 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ae00095f0 con 0x7f7ae40f5410 2023-11-16T01:12:52.226 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ae0007e70 con 0x7f7ae40f5410 2023-11-16T01:12:52.226 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae4060420 con 0x7f7ae40f5410 2023-11-16T01:12:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ae0014ea0 con 0x7f7ae40f5410 2023-11-16T01:12:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.219+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae4109b00 con 0x7f7ae40f5410 2023-11-16T01:12:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.220+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ae0025070 con 0x7f7ae40f5410 2023-11-16T01:12:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.220+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae4059100 con 0x7f7ae40f5410 2023-11-16T01:12:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.220+0000 7f7ad97fa700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 0x7f7acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:52.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.220+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ae0040a50 con 0x7f7ae40f5410 2023-11-16T01:12:52.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.220+0000 7f7adbfff700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 0x7f7acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:52.228 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.221+0000 7f7adbfff700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 0x7f7acc032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7ad40096a0 tx=0x7f7ad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:52.229 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.225+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ae0026350 con 0x7f7ae40f5410 2023-11-16T01:12:52.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.431+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ae4045ad0 con 0x7f7ae40f5410 2023-11-16T01:12:52.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.432+0000 7f7ad97fa700 1 -- 172.21.15.134:0/3012426158 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ae0012330 con 0x7f7ae40f5410 2023-11-16T01:12:52.433 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:52.433 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 msgr2=0x7f7acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 0x7f7acc032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7ad40096a0 tx=0x7f7ad4006b40).stop 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 msgr2=0x7f7ae4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7ae0014000 tx=0x7f7ae000b040).stop 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 shutdown_connections 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7acc02ff60 0x7f7acc032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 --2- 172.21.15.134:0/3012426158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ae40f5410 0x7f7ae4109450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.433+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 >> 172.21.15.134:0/3012426158 conn(0x7f7ae40f0b70 msgr2=0x7f7ae40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:52.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.434+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 shutdown_connections 2023-11-16T01:12:52.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:52.434+0000 7f7aeafa3700 1 -- 172.21.15.134:0/3012426158 wait complete. 2023-11-16T01:12:52.436 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:53.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:52 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:52.433088+0000 mon.smithi133 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.134:0/3012426158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:53.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:53.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:51.970477+0000 mgr.smithi133.xdyttl (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:55.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 -- 172.21.15.134:0/167855111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 msgr2=0x7f30ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 --2- 172.21.15.134:0/167855111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac0f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f30a4009960 tx=0x7f30a4004d10).stop 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 -- 172.21.15.134:0/167855111 shutdown_connections 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 --2- 172.21.15.134:0/167855111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac0f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 -- 172.21.15.134:0/167855111 >> 172.21.15.134:0/167855111 conn(0x7f30ac0f0b70 msgr2=0x7f30ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 -- 172.21.15.134:0/167855111 shutdown_connections 2023-11-16T01:12:55.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.037+0000 7f30b381e700 1 -- 172.21.15.134:0/167855111 wait complete. 2023-11-16T01:12:55.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b381e700 1 Processor -- start 2023-11-16T01:12:55.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b381e700 1 -- start start 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b381e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b381e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac106c10 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b15ba700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b15ba700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33606/0 (socket says 172.21.15.134:33606) 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b15ba700 1 -- 172.21.15.134:0/3641286453 learned_addr learned my addr 172.21.15.134:0/3641286453 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b15ba700 1 -- 172.21.15.134:0/3641286453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4009640 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b15ba700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f30a4004b70 tx=0x7f30a4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a4018070 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30a400b870 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac106e10 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.038+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a401c560 con 0x7f30ac0f5410 2023-11-16T01:12:55.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.039+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1072a0 con 0x7f30ac0f5410 2023-11-16T01:12:55.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.039+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30a4013630 con 0x7f30ac0f5410 2023-11-16T01:12:55.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.040+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac045ad0 con 0x7f30ac0f5410 2023-11-16T01:12:55.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.040+0000 7f30a1ffb700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 0x7f3098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:55.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.040+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30a4041290 con 0x7f30ac0f5410 2023-11-16T01:12:55.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.040+0000 7f30b0db9700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 0x7f3098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:55.043 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.040+0000 7f30b0db9700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 0x7f3098032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f309c0096a0 tx=0x7f309c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:55.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.044+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30a4004810 con 0x7f30ac0f5410 2023-11-16T01:12:55.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.249+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30ac108150 con 0x7f30ac0f5410 2023-11-16T01:12:55.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.250+0000 7f30a1ffb700 1 -- 172.21.15.134:0/3641286453 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30a4012380 con 0x7f30ac0f5410 2023-11-16T01:12:55.252 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:55.252 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:55.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.251+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 msgr2=0x7f3098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:55.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.251+0000 7f30b381e700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 0x7f3098032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f309c0096a0 tx=0x7f309c006b40).stop 2023-11-16T01:12:55.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.251+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 msgr2=0x7f30ac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:55.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.251+0000 7f30b381e700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f30a4004b70 tx=0x7f30a4004d10).stop 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 shutdown_connections 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f309802ff60 0x7f3098032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 --2- 172.21.15.134:0/3641286453 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30ac0f5410 0x7f30ac106720 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 >> 172.21.15.134:0/3641286453 conn(0x7f30ac0f0b70 msgr2=0x7f30ac05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 shutdown_connections 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:55.252+0000 7f30b381e700 1 -- 172.21.15.134:0/3641286453 wait complete. 2023-11-16T01:12:55.254 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:53.970751+0000 mgr.smithi133.xdyttl (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:55.251132+0000 mon.smithi133 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.134:0/3641286453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:56.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:56.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 -- 172.21.15.134:0/417825691 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 msgr2=0x7f81ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 --2- 172.21.15.134:0/417825691 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec0f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f81e4009fb0 tx=0x7f81e4009b30).stop 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 -- 172.21.15.134:0/417825691 shutdown_connections 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 --2- 172.21.15.134:0/417825691 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec0f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 -- 172.21.15.134:0/417825691 >> 172.21.15.134:0/417825691 conn(0x7f81ec0f0b70 msgr2=0x7f81ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 -- 172.21.15.134:0/417825691 shutdown_connections 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.853+0000 7f81f4509700 1 -- 172.21.15.134:0/417825691 wait complete. 2023-11-16T01:12:57.856 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f4509700 1 Processor -- start 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f4509700 1 -- start start 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f4509700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f4509700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec105b20 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f22a5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f22a5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33626/0 (socket says 172.21.15.134:33626) 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f22a5700 1 -- 172.21.15.134:0/403569238 learned_addr learned my addr 172.21.15.134:0/403569238 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f22a5700 1 -- 172.21.15.134:0/403569238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4005040 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81f22a5700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f81e4013510 tx=0x7f81e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.854+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e4009320 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec105d20 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81e4013f00 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e4014d90 con 0x7f81ec0f5410 2023-11-16T01:12:57.857 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec1061b0 con 0x7f81ec0f5410 2023-11-16T01:12:57.858 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81e40263a0 con 0x7f81ec0f5410 2023-11-16T01:12:57.858 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.855+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec045ad0 con 0x7f81ec0f5410 2023-11-16T01:12:57.858 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.856+0000 7f81eaffd700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 0x7f81d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:12:57.859 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.856+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81e4041020 con 0x7f81ec0f5410 2023-11-16T01:12:57.859 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.856+0000 7f81f1aa4700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 0x7f81d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:12:57.859 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.856+0000 7f81f1aa4700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 0x7f81d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f81e00096a0 tx=0x7f81e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:12:57.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:57.860+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81e40126f0 con 0x7f81ec0f5410 2023-11-16T01:12:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:55.971005+0000 mgr.smithi133.xdyttl (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:12:58.067 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.065+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81ec0fae60 con 0x7f81ec0f5410 2023-11-16T01:12:58.067 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.066+0000 7f81eaffd700 1 -- 172.21.15.134:0/403569238 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f81e4012330 con 0x7f81ec0f5410 2023-11-16T01:12:58.067 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:12:58.068 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.067+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 msgr2=0x7f81d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.067+0000 7f81f4509700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 0x7f81d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f81e00096a0 tx=0x7f81e0006b40).stop 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.067+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 msgr2=0x7f81ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.067+0000 7f81f4509700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f81e4013510 tx=0x7f81e400b040).stop 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 shutdown_connections 2023-11-16T01:12:58.069 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f81d402ff60 0x7f81d4032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:58.070 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 --2- 172.21.15.134:0/403569238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81ec0f5410 0x7f81ec105630 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:12:58.070 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 >> 172.21.15.134:0/403569238 conn(0x7f81ec0f0b70 msgr2=0x7f81ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:12:58.070 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 shutdown_connections 2023-11-16T01:12:58.070 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:12:58.068+0000 7f81f4509700 1 -- 172.21.15.134:0/403569238 wait complete. 2023-11-16T01:12:58.070 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:12:59.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:58 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:12:58.067463+0000 mon.smithi133 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.134:0/403569238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:12:59.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:12:59.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:12:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:57.971366+0000 mgr.smithi133.xdyttl (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 -- 172.21.15.134:0/3652519174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 msgr2=0x7f45f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 --2- 172.21.15.134:0/3652519174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f40f6930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f45e4009fb0 tx=0x7f45e4009b30).stop 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 -- 172.21.15.134:0/3652519174 shutdown_connections 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 --2- 172.21.15.134:0/3652519174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f40f6930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 -- 172.21.15.134:0/3652519174 >> 172.21.15.134:0/3652519174 conn(0x7f45f40f1c70 msgr2=0x7f45f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 -- 172.21.15.134:0/3652519174 shutdown_connections 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.654+0000 7f45fa794700 1 -- 172.21.15.134:0/3652519174 wait complete. 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45fa794700 1 Processor -- start 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45fa794700 1 -- start start 2023-11-16T01:13:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45fa794700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45fa794700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4106c20 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45f3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45f3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47338/0 (socket says 172.21.15.134:47338) 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.655+0000 7f45f3fff700 1 -- 172.21.15.134:0/142847635 learned_addr learned my addr 172.21.15.134:0/142847635 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45f3fff700 1 -- 172.21.15.134:0/142847635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4005040 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45f3fff700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f45e4014000 tx=0x7f45e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e40095f0 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f4106e20 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45e4007e70 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e4014ea0 con 0x7f45f40f6510 2023-11-16T01:13:00.659 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.656+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f41072b0 con 0x7f45f40f6510 2023-11-16T01:13:00.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.657+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45e4025070 con 0x7f45f40f6510 2023-11-16T01:13:00.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.657+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f4045ad0 con 0x7f45f40f6510 2023-11-16T01:13:00.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.657+0000 7f45f0ff9700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 0x7f45dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:00.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.657+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45e40409e0 con 0x7f45f40f6510 2023-11-16T01:13:00.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.657+0000 7f45f37fe700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 0x7f45dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:00.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.658+0000 7f45f37fe700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 0x7f45dc032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f45e80096a0 tx=0x7f45e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:00.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.662+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45e400f4e0 con 0x7f45f40f6510 2023-11-16T01:13:00.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.866+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45f40fbf60 con 0x7f45f40f6510 2023-11-16T01:13:00.868 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.867+0000 7f45f0ff9700 1 -- 172.21.15.134:0/142847635 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f45e400f300 con 0x7f45f40f6510 2023-11-16T01:13:00.868 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:00.869 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:00.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.868+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 msgr2=0x7f45dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:00.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.868+0000 7f45fa794700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 0x7f45dc032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f45e80096a0 tx=0x7f45e8006b40).stop 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.868+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 msgr2=0x7f45f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.868+0000 7f45fa794700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f45e4014000 tx=0x7f45e400b040).stop 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 shutdown_connections 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f45dc02ffb0 0x7f45dc032470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 --2- 172.21.15.134:0/142847635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45f40f6510 0x7f45f4106730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 >> 172.21.15.134:0/142847635 conn(0x7f45f40f1c70 msgr2=0x7f45f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 shutdown_connections 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:00.869+0000 7f45fa794700 1 -- 172.21.15.134:0/142847635 wait complete. 2023-11-16T01:13:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:12:59.971619+0000 mgr.smithi133.xdyttl (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:00.868272+0000 mon.smithi133 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.134:0/142847635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:02.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:02.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:03.511 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 -- 172.21.15.134:0/945624733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 msgr2=0x7fc8580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:03.511 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/945624733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc8580f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc854009fb0 tx=0x7fc854009b30).stop 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 -- 172.21.15.134:0/945624733 shutdown_connections 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/945624733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc8580f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 -- 172.21.15.134:0/945624733 >> 172.21.15.134:0/945624733 conn(0x7fc8580f0b70 msgr2=0x7fc8580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 -- 172.21.15.134:0/945624733 shutdown_connections 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 -- 172.21.15.134:0/945624733 wait complete. 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.509+0000 7fc85f2ac700 1 Processor -- start 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85f2ac700 1 -- start start 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85f2ac700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85f2ac700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc858060240 con 0x7fc8580f5410 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85d048700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:03.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85d048700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47356/0 (socket says 172.21.15.134:47356) 2023-11-16T01:13:03.513 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85d048700 1 -- 172.21.15.134:0/3435686713 learned_addr learned my addr 172.21.15.134:0/3435686713 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85d048700 1 -- 172.21.15.134:0/3435686713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc854005040 con 0x7fc8580f5410 2023-11-16T01:13:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.510+0000 7fc85d048700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc854014000 tx=0x7fc85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.511+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8540095f0 con 0x7fc8580f5410 2023-11-16T01:13:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.511+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc854007e70 con 0x7fc8580f5410 2023-11-16T01:13:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.511+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc858060440 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.511+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc854014ea0 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.511+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc858109ae0 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc854025070 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc858045ad0 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc84dffb700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 0x7fc844032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8540409a0 con 0x7fc8580f5410 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc85c847700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 0x7fc844032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:03.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.512+0000 7fc85c847700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 0x7fc844032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc8480096a0 tx=0x7fc848006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:03.518 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.517+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc85400f4e0 con 0x7fc8580f5410 2023-11-16T01:13:03.722 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.720+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8580fae60 con 0x7fc8580f5410 2023-11-16T01:13:03.723 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.721+0000 7fc84dffb700 1 -- 172.21.15.134:0/3435686713 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc85400f300 con 0x7fc8580f5410 2023-11-16T01:13:03.723 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:03.723 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:03.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 msgr2=0x7fc844032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:03.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 0x7fc844032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc8480096a0 tx=0x7fc848006b40).stop 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 msgr2=0x7fc858109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc854014000 tx=0x7fc85400b040).stop 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 shutdown_connections 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc84402ffb0 0x7fc844032470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 --2- 172.21.15.134:0/3435686713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8580f5410 0x7fc858109430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 >> 172.21.15.134:0/3435686713 conn(0x7fc8580f0b70 msgr2=0x7fc8580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 shutdown_connections 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:03.723+0000 7fc85f2ac700 1 -- 172.21.15.134:0/3435686713 wait complete. 2023-11-16T01:13:03.725 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:01.971897+0000 mgr.smithi133.xdyttl (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:05.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:04 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:03.722862+0000 mon.smithi133 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.134:0/3435686713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:05.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:05.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:03.972170+0000 mgr.smithi133.xdyttl (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:06.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 -- 172.21.15.134:0/1325230025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 msgr2=0x7f7a2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 --2- 172.21.15.134:0/1325230025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c0f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f7a1c009fb0 tx=0x7f7a1c009b30).stop 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 -- 172.21.15.134:0/1325230025 shutdown_connections 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 --2- 172.21.15.134:0/1325230025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c0f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 -- 172.21.15.134:0/1325230025 >> 172.21.15.134:0/1325230025 conn(0x7f7a2c0f1c70 msgr2=0x7f7a2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 -- 172.21.15.134:0/1325230025 shutdown_connections 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.315+0000 7f7a327c8700 1 -- 172.21.15.134:0/1325230025 wait complete. 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a327c8700 1 Processor -- start 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a327c8700 1 -- start start 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a327c8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a327c8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c106c20 con 0x7f7a2c0f6510 2023-11-16T01:13:06.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a2bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a2bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47374/0 (socket says 172.21.15.134:47374) 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.316+0000 7f7a2bfff700 1 -- 172.21.15.134:0/2927882197 learned_addr learned my addr 172.21.15.134:0/2927882197 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a2bfff700 1 -- 172.21.15.134:0/2927882197 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c005040 con 0x7f7a2c0f6510 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a2bfff700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7a1c013510 tx=0x7f7a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1c009320 con 0x7f7a2c0f6510 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c106e20 con 0x7f7a2c0f6510 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a1c013f00 con 0x7f7a2c0f6510 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1c014d90 con 0x7f7a2c0f6510 2023-11-16T01:13:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.317+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1072b0 con 0x7f7a2c0f6510 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.318+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a1c014420 con 0x7f7a2c0f6510 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.318+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c045ad0 con 0x7f7a2c0f6510 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.318+0000 7f7a28ff9700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 0x7f7a14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.318+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a1c0410c0 con 0x7f7a2c0f6510 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.318+0000 7f7a2b7fe700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 0x7f7a14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:06.321 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.319+0000 7f7a2b7fe700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 0x7f7a14032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7a200096a0 tx=0x7f7a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:06.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.323+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a1c0095f0 con 0x7f7a2c0f6510 2023-11-16T01:13:06.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.529+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a2c0fbf60 con 0x7f7a2c0f6510 2023-11-16T01:13:06.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.529+0000 7f7a28ff9700 1 -- 172.21.15.134:0/2927882197 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a1c01db70 con 0x7f7a2c0f6510 2023-11-16T01:13:06.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:06.531 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 msgr2=0x7f7a14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 0x7f7a14032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7a200096a0 tx=0x7f7a20006b40).stop 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 msgr2=0x7f7a2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7a1c013510 tx=0x7f7a1c00b040).stop 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 shutdown_connections 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7a1402ff60 0x7f7a14032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 --2- 172.21.15.134:0/2927882197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7a2c0f6510 0x7f7a2c106730 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 >> 172.21.15.134:0/2927882197 conn(0x7f7a2c0f1c70 msgr2=0x7f7a2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 shutdown_connections 2023-11-16T01:13:06.533 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:06.531+0000 7f7a327c8700 1 -- 172.21.15.134:0/2927882197 wait complete. 2023-11-16T01:13:06.534 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:07.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:06.530967+0000 mon.smithi133 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.134:0/2927882197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:07.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:07.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:05.972489+0000 mgr.smithi133.xdyttl (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:09.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 -- 172.21.15.134:0/3003254119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 msgr2=0x7f6a600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:09.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 --2- 172.21.15.134:0/3003254119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a600f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f6a54009fb0 tx=0x7f6a54009b30).stop 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 -- 172.21.15.134:0/3003254119 shutdown_connections 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 --2- 172.21.15.134:0/3003254119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a600f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 -- 172.21.15.134:0/3003254119 >> 172.21.15.134:0/3003254119 conn(0x7f6a600f0b70 msgr2=0x7f6a600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 -- 172.21.15.134:0/3003254119 shutdown_connections 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.190+0000 7f6a653c4700 1 -- 172.21.15.134:0/3003254119 wait complete. 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a653c4700 1 Processor -- start 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a653c4700 1 -- start start 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a653c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a653c4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60060280 con 0x7f6a600f5410 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a5effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a5effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:47386/0 (socket says 172.21.15.134:47386) 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a5effd700 1 -- 172.21.15.134:0/504314107 learned_addr learned my addr 172.21.15.134:0/504314107 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a5effd700 1 -- 172.21.15.134:0/504314107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54005040 con 0x7f6a600f5410 2023-11-16T01:13:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.191+0000 7f6a5effd700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6a54013510 tx=0x7f6a5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a54009320 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a54013f00 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a60060480 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a54014d90 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a60109af0 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.192+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a540263a0 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.193+0000 7f6a477fe700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 0x7f6a48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.193+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a540410b0 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.193+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a60045ad0 con 0x7f6a600f5410 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.193+0000 7f6a5e7fc700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 0x7f6a48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.193+0000 7f6a5e7fc700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 0x7f6a48032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6a500096a0 tx=0x7f6a50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.198+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a54012920 con 0x7f6a600f5410 2023-11-16T01:13:09.407 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.406+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a600faf50 con 0x7f6a600f5410 2023-11-16T01:13:09.408 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.406+0000 7f6a477fe700 1 -- 172.21.15.134:0/504314107 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a5402b020 con 0x7f6a600f5410 2023-11-16T01:13:09.408 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:09.408 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:09.410 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 msgr2=0x7f6a48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:09.410 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 0x7f6a48032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6a500096a0 tx=0x7f6a50006b40).stop 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 msgr2=0x7f6a601093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6a54013510 tx=0x7f6a5400b040).stop 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 shutdown_connections 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a4802ffb0 0x7f6a48032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 --2- 172.21.15.134:0/504314107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a600f5410 0x7f6a601093c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 >> 172.21.15.134:0/504314107 conn(0x7f6a600f0b70 msgr2=0x7f6a600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 shutdown_connections 2023-11-16T01:13:09.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:09.408+0000 7f6a653c4700 1 -- 172.21.15.134:0/504314107 wait complete. 2023-11-16T01:13:09.412 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:07.972812+0000 mgr.smithi133.xdyttl (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:09.407840+0000 mon.smithi133 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.134:0/504314107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:10.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:10.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:11.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- 172.21.15.134:0/56296070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 msgr2=0x7f0c100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:11.969 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 --2- 172.21.15.134:0/56296070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c100f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f0c00009fb0 tx=0x7f0c00009b30).stop 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- 172.21.15.134:0/56296070 shutdown_connections 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 --2- 172.21.15.134:0/56296070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c100f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- 172.21.15.134:0/56296070 >> 172.21.15.134:0/56296070 conn(0x7f0c100f0b70 msgr2=0x7f0c100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- 172.21.15.134:0/56296070 shutdown_connections 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- 172.21.15.134:0/56296070 wait complete. 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 Processor -- start 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.967+0000 7f0c184c7700 1 -- start start 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c184c7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c184c7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c10060240 con 0x7f0c100f5410 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c16263700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:11.970 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c16263700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42936/0 (socket says 172.21.15.134:42936) 2023-11-16T01:13:11.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c16263700 1 -- 172.21.15.134:0/1191017893 learned_addr learned my addr 172.21.15.134:0/1191017893 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:11.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c16263700 1 -- 172.21.15.134:0/1191017893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00005040 con 0x7f0c100f5410 2023-11-16T01:13:11.971 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c16263700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f0c00013510 tx=0x7f0c0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00009320 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c00013f00 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c10060440 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.968+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00014d90 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.969+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c10109ab0 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.969+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c000095f0 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.969+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c10045ad0 con 0x7f0c100f5410 2023-11-16T01:13:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.969+0000 7f0c06ffd700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:11.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.969+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c000410c0 con 0x7f0c100f5410 2023-11-16T01:13:11.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.970+0000 7f0c15a62700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:11.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.970+0000 7f0c15a62700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 0x7f0bfc032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0c0c0096a0 tx=0x7f0c0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:11.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:11.975+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c00014420 con 0x7f0c100f5410 2023-11-16T01:13:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:09.973162+0000 mgr.smithi133.xdyttl (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:12.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.180+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c100fae60 con 0x7f0c100f5410 2023-11-16T01:13:12.181 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.180+0000 7f0c06ffd700 1 -- 172.21.15.134:0/1191017893 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c0001d7f0 con 0x7f0c100f5410 2023-11-16T01:13:12.181 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:12.181 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:12.182 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 msgr2=0x7f0bfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:12.182 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 0x7f0bfc032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0c0c0096a0 tx=0x7f0c0c006b40).stop 2023-11-16T01:13:12.182 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 msgr2=0x7f0c10109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f0c00013510 tx=0x7f0c0000b040).stop 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 shutdown_connections 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 --2- 172.21.15.134:0/1191017893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0c100f5410 0x7f0c10109400 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 >> 172.21.15.134:0/1191017893 conn(0x7f0c100f0b70 msgr2=0x7f0c100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 shutdown_connections 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:12.182+0000 7f0c184c7700 1 -- 172.21.15.134:0/1191017893 wait complete. 2023-11-16T01:13:12.183 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:13.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:12 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:12.180879+0000 mon.smithi133 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.134:0/1191017893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:13.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:13.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:11.973431+0000 mgr.smithi133.xdyttl (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:14.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 -- 172.21.15.134:0/990654786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 msgr2=0x7ff4200f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 --2- 172.21.15.134:0/990654786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4200f9190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7ff410009fb0 tx=0x7ff410009b30).stop 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 -- 172.21.15.134:0/990654786 shutdown_connections 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 --2- 172.21.15.134:0/990654786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4200f9190 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 -- 172.21.15.134:0/990654786 >> 172.21.15.134:0/990654786 conn(0x7ff4200f0b70 msgr2=0x7ff4200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 -- 172.21.15.134:0/990654786 shutdown_connections 2023-11-16T01:13:14.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.760+0000 7ff427b32700 1 -- 172.21.15.134:0/990654786 wait complete. 2023-11-16T01:13:14.762 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff427b32700 1 Processor -- start 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff427b32700 1 -- start start 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff427b32700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff427b32700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff420106bd0 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff4258ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff4258ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42958/0 (socket says 172.21.15.134:42958) 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.761+0000 7ff4258ce700 1 -- 172.21.15.134:0/437074070 learned_addr learned my addr 172.21.15.134:0/437074070 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff4258ce700 1 -- 172.21.15.134:0/437074070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410005040 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff4258ce700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff410004dd0 tx=0x7ff410009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff410018070 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff410013410 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff41001c3c0 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff420106dd0 con 0x7ff4200601e0 2023-11-16T01:13:14.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.762+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff420107260 con 0x7ff4200601e0 2023-11-16T01:13:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff41001c520 con 0x7ff4200601e0 2023-11-16T01:13:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff4167fc700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 0x7ff40c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff410041670 con 0x7ff4200601e0 2023-11-16T01:13:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420045ad0 con 0x7ff4200601e0 2023-11-16T01:13:14.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff4250cd700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 0x7ff40c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:14.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.763+0000 7ff4250cd700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 0x7ff40c032470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff41c0096a0 tx=0x7ff41c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:14.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.767+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff41001c920 con 0x7ff4200601e0 2023-11-16T01:13:14.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.973+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4200fc050 con 0x7ff4200601e0 2023-11-16T01:13:14.974 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.974+0000 7ff4167fc700 1 -- 172.21.15.134:0/437074070 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff41001b9c0 con 0x7ff4200601e0 2023-11-16T01:13:14.974 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:14.974 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.975+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 msgr2=0x7ff40c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.975+0000 7ff427b32700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 0x7ff40c032470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff41c0096a0 tx=0x7ff41c006b40).stop 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.975+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 msgr2=0x7ff4201066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.975+0000 7ff427b32700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff410004dd0 tx=0x7ff410009b30).stop 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 shutdown_connections 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff40c02ffb0 0x7ff40c032470 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:14.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 --2- 172.21.15.134:0/437074070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4200601e0 0x7ff4201066e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 >> 172.21.15.134:0/437074070 conn(0x7ff4200f0b70 msgr2=0x7ff4200f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 shutdown_connections 2023-11-16T01:13:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:14.976+0000 7ff427b32700 1 -- 172.21.15.134:0/437074070 wait complete. 2023-11-16T01:13:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:13.973778+0000 mgr.smithi133.xdyttl (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:14.974405+0000 mon.smithi133 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.134:0/437074070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:16.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:16.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 -- 172.21.15.134:0/2552976538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 msgr2=0x7fd6900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 --2- 172.21.15.134:0/2552976538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd6900f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fd680009fb0 tx=0x7fd680009b30).stop 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 -- 172.21.15.134:0/2552976538 shutdown_connections 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 --2- 172.21.15.134:0/2552976538 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd6900f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 -- 172.21.15.134:0/2552976538 >> 172.21.15.134:0/2552976538 conn(0x7fd6900f1c70 msgr2=0x7fd6900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 -- 172.21.15.134:0/2552976538 shutdown_connections 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.557+0000 7fd6973a0700 1 -- 172.21.15.134:0/2552976538 wait complete. 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd6973a0700 1 Processor -- start 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd6973a0700 1 -- start start 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd6973a0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd6973a0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd690106c20 con 0x7fd6900f6510 2023-11-16T01:13:17.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd69513c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:17.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd69513c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:42978/0 (socket says 172.21.15.134:42978) 2023-11-16T01:13:17.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd69513c700 1 -- 172.21.15.134:0/563957639 learned_addr learned my addr 172.21.15.134:0/563957639 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:17.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.558+0000 7fd69513c700 1 -- 172.21.15.134:0/563957639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd680005040 con 0x7fd6900f6510 2023-11-16T01:13:17.560 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd69513c700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd680013510 tx=0x7fd68000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd680009320 con 0x7fd6900f6510 2023-11-16T01:13:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd690106e20 con 0x7fd6900f6510 2023-11-16T01:13:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd680013f00 con 0x7fd6900f6510 2023-11-16T01:13:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd680014d90 con 0x7fd6900f6510 2023-11-16T01:13:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.559+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6901072b0 con 0x7fd6900f6510 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd680014420 con 0x7fd6900f6510 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd690045ad0 con 0x7fd6900f6510 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd685ffb700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd680041110 con 0x7fd6900f6510 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd69493b700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.560+0000 7fd69493b700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 0x7fd67c032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.565+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6800095f0 con 0x7fd6900f6510 2023-11-16T01:13:17.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.771+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6900fbf60 con 0x7fd6900f6510 2023-11-16T01:13:17.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.772+0000 7fd685ffb700 1 -- 172.21.15.134:0/563957639 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd68001db70 con 0x7fd6900f6510 2023-11-16T01:13:17.773 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:17.773 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:17.774 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 msgr2=0x7fd67c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 0x7fd67c032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).stop 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 msgr2=0x7fd690106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd680013510 tx=0x7fd68000b040).stop 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 shutdown_connections 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 --2- 172.21.15.134:0/563957639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6900f6510 0x7fd690106730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:17.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 >> 172.21.15.134:0/563957639 conn(0x7fd6900f1c70 msgr2=0x7fd6900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:17.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 shutdown_connections 2023-11-16T01:13:17.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:17.774+0000 7fd6973a0700 1 -- 172.21.15.134:0/563957639 wait complete. 2023-11-16T01:13:17.776 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:15.974111+0000 mgr.smithi133.xdyttl (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:19.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:17.772789+0000 mon.smithi133 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.134:0/563957639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:19.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:19.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:17.974465+0000 mgr.smithi133.xdyttl (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 -- 172.21.15.134:0/1833357467 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 msgr2=0x7f13340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 --2- 172.21.15.134:0/1833357467 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13340f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1328009fb0 tx=0x7f1328009b30).stop 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 -- 172.21.15.134:0/1833357467 shutdown_connections 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 --2- 172.21.15.134:0/1833357467 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13340f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 -- 172.21.15.134:0/1833357467 >> 172.21.15.134:0/1833357467 conn(0x7f13340f0b70 msgr2=0x7f13340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 -- 172.21.15.134:0/1833357467 shutdown_connections 2023-11-16T01:13:20.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.444+0000 7f133c52e700 1 -- 172.21.15.134:0/1833357467 wait complete. 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133c52e700 1 Processor -- start 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133c52e700 1 -- start start 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133c52e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133c52e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334106bd0 con 0x7f13340f5410 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133a2ca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133a2ca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59258/0 (socket says 172.21.15.134:59258) 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133a2ca700 1 -- 172.21.15.134:0/4284555823 learned_addr learned my addr 172.21.15.134:0/4284555823 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133a2ca700 1 -- 172.21.15.134:0/4284555823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328005040 con 0x7f13340f5410 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133a2ca700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1328013510 tx=0x7f132800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328009320 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1328013f00 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.445+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1334106dd0 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.446+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328014d90 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.446+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1334107260 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.446+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13280263a0 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.446+0000 7f1326ffd700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 0x7f1320032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.446+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13280410c0 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.447+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334059100 con 0x7f13340f5410 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.447+0000 7f1339ac9700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 0x7f1320032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:20.448 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.447+0000 7f1339ac9700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 0x7f1320032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f13300096a0 tx=0x7f1330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:20.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.451+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1328012920 con 0x7f13340f5410 2023-11-16T01:13:20.654 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.654+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13340fae60 con 0x7f13340f5410 2023-11-16T01:13:20.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.654+0000 7f1326ffd700 1 -- 172.21.15.134:0/4284555823 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f132802b020 con 0x7f13340f5410 2023-11-16T01:13:20.655 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:20.655 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:20.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 msgr2=0x7f1320032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 0x7f1320032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f13300096a0 tx=0x7f1330006b40).stop 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 msgr2=0x7f13341066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1328013510 tx=0x7f132800b040).stop 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 shutdown_connections 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f132002ff60 0x7f1320032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 --2- 172.21.15.134:0/4284555823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13340f5410 0x7f13341066e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 >> 172.21.15.134:0/4284555823 conn(0x7f13340f0b70 msgr2=0x7f133405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 shutdown_connections 2023-11-16T01:13:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:20.656+0000 7f133c52e700 1 -- 172.21.15.134:0/4284555823 wait complete. 2023-11-16T01:13:20.658 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:21.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:20.655045+0000 mon.smithi133 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.134:0/4284555823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:22.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:19.974777+0000 mgr.smithi133.xdyttl (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:22.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:22.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:23.215 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 -- 172.21.15.134:0/3401597685 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 msgr2=0x7fca080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:23.215 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3401597685 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca080f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fc9fc009fb0 tx=0x7fc9fc009b30).stop 2023-11-16T01:13:23.215 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 -- 172.21.15.134:0/3401597685 shutdown_connections 2023-11-16T01:13:23.215 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3401597685 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca080f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:23.215 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 -- 172.21.15.134:0/3401597685 >> 172.21.15.134:0/3401597685 conn(0x7fca080f1c70 msgr2=0x7fca080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 -- 172.21.15.134:0/3401597685 shutdown_connections 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.213+0000 7fca0d92b700 1 -- 172.21.15.134:0/3401597685 wait complete. 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca0d92b700 1 Processor -- start 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca0d92b700 1 -- start start 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca0d92b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca0d92b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca08069e20 con 0x7fca080f6510 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca06ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca06ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59274/0 (socket says 172.21.15.134:59274) 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca06ffd700 1 -- 172.21.15.134:0/3618596448 learned_addr learned my addr 172.21.15.134:0/3618596448 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.214+0000 7fca06ffd700 1 -- 172.21.15.134:0/3618596448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc005040 con 0x7fca080f6510 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fca06ffd700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc9fc013510 tx=0x7fc9fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:23.216 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc009320 con 0x7fca080f6510 2023-11-16T01:13:23.217 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9fc013f00 con 0x7fca080f6510 2023-11-16T01:13:23.217 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca08069fc0 con 0x7fca080f6510 2023-11-16T01:13:23.217 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9fc014d90 con 0x7fca080f6510 2023-11-16T01:13:23.217 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.215+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0806a450 con 0x7fca080f6510 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9fc014420 con 0x7fca080f6510 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca08059100 con 0x7fca080f6510 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fc9ef7fe700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 0x7fc9f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9fc041110 con 0x7fca080f6510 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fca067fc700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 0x7fc9f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:23.218 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.216+0000 7fca067fc700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 0x7fc9f0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc9f80096a0 tx=0x7fc9f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:23.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.220+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9fc0095f0 con 0x7fca080f6510 2023-11-16T01:13:23.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.426+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca080fbf60 con 0x7fca080f6510 2023-11-16T01:13:23.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.427+0000 7fc9ef7fe700 1 -- 172.21.15.134:0/3618596448 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc9fc01db70 con 0x7fca080f6510 2023-11-16T01:13:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:23.428 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:23.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 msgr2=0x7fc9f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:23.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 0x7fc9f0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc9f80096a0 tx=0x7fc9f8006b40).stop 2023-11-16T01:13:23.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 msgr2=0x7fca0806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:23.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc9fc013510 tx=0x7fc9fc00b040).stop 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 shutdown_connections 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc9f002ff60 0x7fc9f0032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 --2- 172.21.15.134:0/3618596448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca080f6510 0x7fca0806b9a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 >> 172.21.15.134:0/3618596448 conn(0x7fca080f1c70 msgr2=0x7fca080f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 shutdown_connections 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:23.429+0000 7fca0d92b700 1 -- 172.21.15.134:0/3618596448 wait complete. 2023-11-16T01:13:23.430 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:21.975161+0000 mgr.smithi133.xdyttl (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:23.427548+0000 mon.smithi133 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.134:0/3618596448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:24.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:24.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:23.975520+0000 mgr.smithi133.xdyttl (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:26.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 -- 172.21.15.134:0/3619674823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 msgr2=0x7fce740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:26.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 --2- 172.21.15.134:0/3619674823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce740f4b60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fce68009fb0 tx=0x7fce68009b30).stop 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 -- 172.21.15.134:0/3619674823 shutdown_connections 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 --2- 172.21.15.134:0/3619674823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce740f4b60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 -- 172.21.15.134:0/3619674823 >> 172.21.15.134:0/3619674823 conn(0x7fce740f0260 msgr2=0x7fce740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 -- 172.21.15.134:0/3619674823 shutdown_connections 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.030+0000 7fce7abe3700 1 -- 172.21.15.134:0/3619674823 wait complete. 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.031+0000 7fce7abe3700 1 Processor -- start 2023-11-16T01:13:26.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.031+0000 7fce7abe3700 1 -- start start 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.031+0000 7fce7abe3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.031+0000 7fce7abe3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74106bc0 con 0x7fce740f4740 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.031+0000 7fce7897f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7897f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59304/0 (socket says 172.21.15.134:59304) 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7897f700 1 -- 172.21.15.134:0/1829105423 learned_addr learned my addr 172.21.15.134:0/1829105423 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7897f700 1 -- 172.21.15.134:0/1829105423 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68005040 con 0x7fce740f4740 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7897f700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fce68013510 tx=0x7fce6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce68009320 con 0x7fce740f4740 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce74106dc0 con 0x7fce740f4740 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce68013f00 con 0x7fce740f4740 2023-11-16T01:13:26.034 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce68014d90 con 0x7fce740f4740 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.032+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce74107250 con 0x7fce740f4740 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce68014420 con 0x7fce740f4740 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce717fa700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 0x7fce64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce74045ad0 con 0x7fce740f4740 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce68041000 con 0x7fce740f4740 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce73fff700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 0x7fce64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:26.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.033+0000 7fce73fff700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 0x7fce64032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fce600096a0 tx=0x7fce60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:26.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.038+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce680095f0 con 0x7fce740f4740 2023-11-16T01:13:26.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.240+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce74108100 con 0x7fce740f4740 2023-11-16T01:13:26.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.241+0000 7fce717fa700 1 -- 172.21.15.134:0/1829105423 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce6801db70 con 0x7fce740f4740 2023-11-16T01:13:26.242 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:26.242 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:26.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.242+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 msgr2=0x7fce64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:26.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.242+0000 7fce7abe3700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 0x7fce64032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fce600096a0 tx=0x7fce60006b40).stop 2023-11-16T01:13:26.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.242+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 msgr2=0x7fce741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:26.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.242+0000 7fce7abe3700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fce68013510 tx=0x7fce6800b040).stop 2023-11-16T01:13:26.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 shutdown_connections 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce6402ff60 0x7fce64032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 --2- 172.21.15.134:0/1829105423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce740f4740 0x7fce741066d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 >> 172.21.15.134:0/1829105423 conn(0x7fce740f0260 msgr2=0x7fce740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 shutdown_connections 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:26.243+0000 7fce7abe3700 1 -- 172.21.15.134:0/1829105423 wait complete. 2023-11-16T01:13:26.244 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:27.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:26 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:26.241507+0000 mon.smithi133 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.134:0/1829105423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:27.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:27.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:25.975813+0000 mgr.smithi133.xdyttl (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 -- 172.21.15.134:0/278184006 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0f6510 msgr2=0x7f5b6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 --2- 172.21.15.134:0/278184006 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0f6510 0x7f5b6c0f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f5b54009fb0 tx=0x7f5b54009b30).stop 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 -- 172.21.15.134:0/278184006 shutdown_connections 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 --2- 172.21.15.134:0/278184006 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c0f6510 0x7f5b6c0f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 -- 172.21.15.134:0/278184006 >> 172.21.15.134:0/278184006 conn(0x7f5b6c0f1c70 msgr2=0x7f5b6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 -- 172.21.15.134:0/278184006 shutdown_connections 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.855+0000 7f5b716b9700 1 -- 172.21.15.134:0/278184006 wait complete. 2023-11-16T01:13:28.860 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b716b9700 1 Processor -- start 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b716b9700 1 -- start start 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b716b9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b716b9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c107100 con 0x7f5b6c1067f0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b6affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b6affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59322/0 (socket says 172.21.15.134:59322) 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b6affd700 1 -- 172.21.15.134:0/1942743742 learned_addr learned my addr 172.21.15.134:0/1942743742 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b6affd700 1 -- 172.21.15.134:0/1942743742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54005040 con 0x7f5b6c1067f0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.856+0000 7f5b6affd700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5b54009f80 tx=0x7f5b54009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b540136c0 con 0x7f5b6c1067f0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b54013e70 con 0x7f5b6c1067f0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c107300 con 0x7f5b6c1067f0 2023-11-16T01:13:28.861 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54014420 con 0x7f5b6c1067f0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c107e50 con 0x7f5b6c1067f0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b54025060 con 0x7f5b6c1067f0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.857+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c059100 con 0x7f5b6c1067f0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.858+0000 7f5b637fe700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 0x7f5b58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.858+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b54040f30 con 0x7f5b6c1067f0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.858+0000 7f5b6a7fc700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 0x7f5b58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:28.862 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.858+0000 7f5b6a7fc700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 0x7f5b58032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5b5c0096a0 tx=0x7f5b5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:28.863 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:28.862+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b5401cb30 con 0x7f5b6c1067f0 2023-11-16T01:13:29.067 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.066+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b6c045ad0 con 0x7f5b6c1067f0 2023-11-16T01:13:29.068 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.067+0000 7f5b637fe700 1 -- 172.21.15.134:0/1942743742 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b5402d2d0 con 0x7f5b6c1067f0 2023-11-16T01:13:29.068 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:29.069 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:29.070 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 msgr2=0x7f5b58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 0x7f5b58032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f5b5c0096a0 tx=0x7f5b5c006b40).stop 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 msgr2=0x7f5b6c106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5b54009f80 tx=0x7f5b54009b30).stop 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 shutdown_connections 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5b5802ff60 0x7f5b58032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 --2- 172.21.15.134:0/1942743742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b6c1067f0 0x7f5b6c106c10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 >> 172.21.15.134:0/1942743742 conn(0x7f5b6c0f1c70 msgr2=0x7f5b6c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.068+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 shutdown_connections 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:29.069+0000 7f5b716b9700 1 -- 172.21.15.134:0/1942743742 wait complete. 2023-11-16T01:13:29.071 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:30.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:27.976121+0000 mgr.smithi133.xdyttl (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:30.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:29.068356+0000 mon.smithi133 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.134:0/1942743742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:30.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:30.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:30.075015+0000 mon.smithi133 (mon.0) 218 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:13:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:30.076145+0000 mon.smithi133 (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 -- 172.21.15.134:0/1447492021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 msgr2=0x7f9d200f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 --2- 172.21.15.134:0/1447492021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200f6db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f9d10009fb0 tx=0x7f9d10009b30).stop 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 -- 172.21.15.134:0/1447492021 shutdown_connections 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 --2- 172.21.15.134:0/1447492021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200f6db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 -- 172.21.15.134:0/1447492021 >> 172.21.15.134:0/1447492021 conn(0x7f9d200f0260 msgr2=0x7f9d200f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 -- 172.21.15.134:0/1447492021 shutdown_connections 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.742+0000 7f9d265bc700 1 -- 172.21.15.134:0/1447492021 wait complete. 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d265bc700 1 Processor -- start 2023-11-16T01:13:31.745 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d265bc700 1 -- start start 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d265bc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d265bc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200fb820 con 0x7f9d200f6990 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d1ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35066/0 (socket says 172.21.15.134:35066) 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d1ffff700 1 -- 172.21.15.134:0/4172241328 learned_addr learned my addr 172.21.15.134:0/4172241328 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.743+0000 7f9d1ffff700 1 -- 172.21.15.134:0/4172241328 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10005040 con 0x7f9d200f6990 2023-11-16T01:13:31.746 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d1ffff700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f9d10013510 tx=0x7f9d1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d10009320 con 0x7f9d200f6990 2023-11-16T01:13:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d10013f00 con 0x7f9d200f6990 2023-11-16T01:13:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d200fb9c0 con 0x7f9d200f6990 2023-11-16T01:13:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d10014d90 con 0x7f9d200f6990 2023-11-16T01:13:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d200fbe50 con 0x7f9d200f6990 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.744+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d10014420 con 0x7f9d200f6990 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.745+0000 7f9d1cff9700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 0x7f9d08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.745+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d20059100 con 0x7f9d200f6990 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.745+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d100410c0 con 0x7f9d200f6990 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.745+0000 7f9d1f7fe700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 0x7f9d08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:31.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.745+0000 7f9d1f7fe700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 0x7f9d08032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:31.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.750+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d1006d2b0 con 0x7f9d200f6990 2023-11-16T01:13:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.952+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d200fae60 con 0x7f9d200f6990 2023-11-16T01:13:31.954 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.953+0000 7f9d1cff9700 1 -- 172.21.15.134:0/4172241328 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d100095f0 con 0x7f9d200f6990 2023-11-16T01:13:31.954 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:31.954 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 msgr2=0x7f9d08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 0x7f9d08032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).stop 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 msgr2=0x7f9d200fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f9d10013510 tx=0x7f9d1000b040).stop 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 shutdown_connections 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9d0802ff60 0x7f9d08032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 --2- 172.21.15.134:0/4172241328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d200f6990 0x7f9d200fd3a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 >> 172.21.15.134:0/4172241328 conn(0x7f9d200f0260 msgr2=0x7f9d200f4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 shutdown_connections 2023-11-16T01:13:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:31.954+0000 7f9d265bc700 1 -- 172.21.15.134:0/4172241328 wait complete. 2023-11-16T01:13:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:32.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:29.976482+0000 mgr.smithi133.xdyttl (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:33.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:31.954393+0000 mon.smithi133 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.134:0/4172241328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:33.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:33.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:31.976820+0000 mgr.smithi133.xdyttl (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- 172.21.15.134:0/1703586427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 msgr2=0x7f99280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 --2- 172.21.15.134:0/1703586427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99280f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9918009fb0 tx=0x7f9918009b30).stop 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- 172.21.15.134:0/1703586427 shutdown_connections 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 --2- 172.21.15.134:0/1703586427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99280f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- 172.21.15.134:0/1703586427 >> 172.21.15.134:0/1703586427 conn(0x7f99280f09a0 msgr2=0x7f99280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- 172.21.15.134:0/1703586427 shutdown_connections 2023-11-16T01:13:34.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- 172.21.15.134:0/1703586427 wait complete. 2023-11-16T01:13:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 Processor -- start 2023-11-16T01:13:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.535+0000 7f992c835700 1 -- start start 2023-11-16T01:13:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f992c835700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:34.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f992c835700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928060240 con 0x7f99280f51a0 2023-11-16T01:13:34.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f99277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f99277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35094/0 (socket says 172.21.15.134:35094) 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f99277fe700 1 -- 172.21.15.134:0/3707288261 learned_addr learned my addr 172.21.15.134:0/3707288261 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f99277fe700 1 -- 172.21.15.134:0/3707288261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918005040 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.536+0000 7f99277fe700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9918009830 tx=0x7f991800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918009320 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9928060440 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99180136c0 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f991801ee90 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9928109860 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9918014420 con 0x7f99280f51a0 2023-11-16T01:13:34.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:34.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9918041210 con 0x7f99280f51a0 2023-11-16T01:13:34.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.537+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928045ad0 con 0x7f99280f51a0 2023-11-16T01:13:34.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.538+0000 7f9926ffd700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:34.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.538+0000 7f9926ffd700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 0x7f99100323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f991c0096a0 tx=0x7f991c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:34.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.542+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f991801c320 con 0x7f99280f51a0 2023-11-16T01:13:34.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.747+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9928109f10 con 0x7f99280f51a0 2023-11-16T01:13:34.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.747+0000 7f990ffff700 1 -- 172.21.15.134:0/3707288261 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f991800fe00 con 0x7f99280f51a0 2023-11-16T01:13:34.749 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:34.749 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 msgr2=0x7f99100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 0x7f99100323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f991c0096a0 tx=0x7f991c006b40).stop 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 msgr2=0x7f99281091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9918009830 tx=0x7f991800b040).stop 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 shutdown_connections 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 --2- 172.21.15.134:0/3707288261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99280f51a0 0x7f99281091b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 >> 172.21.15.134:0/3707288261 conn(0x7f99280f09a0 msgr2=0x7f99280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 shutdown_connections 2023-11-16T01:13:34.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:34.749+0000 7f992c835700 1 -- 172.21.15.134:0/3707288261 wait complete. 2023-11-16T01:13:34.752 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:33.977138+0000 mgr.smithi133.xdyttl (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:34.748790+0000 mon.smithi133 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.134:0/3707288261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:36.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:36.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:36.990 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:36.432742+0000 mon.smithi133 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-16T01:13:36.990 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:36.435358+0000 mon.smithi133 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-16T01:13:36.990 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:36.436139+0000 mon.smithi133 (mon.0) 224 : audit [DBG] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T01:13:37.349 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 -- 172.21.15.134:0/3961285216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 msgr2=0x7fafd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:37.349 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 --2- 172.21.15.134:0/3961285216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fafc4009fb0 tx=0x7fafc4009b30).stop 2023-11-16T01:13:37.349 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 -- 172.21.15.134:0/3961285216 shutdown_connections 2023-11-16T01:13:37.349 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 --2- 172.21.15.134:0/3961285216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 -- 172.21.15.134:0/3961285216 >> 172.21.15.134:0/3961285216 conn(0x7fafd00f0b70 msgr2=0x7fafd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 -- 172.21.15.134:0/3961285216 shutdown_connections 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.347+0000 7fafd70e8700 1 -- 172.21.15.134:0/3961285216 wait complete. 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd70e8700 1 Processor -- start 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd70e8700 1 -- start start 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd70e8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:37.350 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd70e8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00fb820 con 0x7fafd00f5410 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd4e84700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd4e84700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35124/0 (socket says 172.21.15.134:35124) 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd4e84700 1 -- 172.21.15.134:0/4292646057 learned_addr learned my addr 172.21.15.134:0/4292646057 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.348+0000 7fafd4e84700 1 -- 172.21.15.134:0/4292646057 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4005040 con 0x7fafd00f5410 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafd4e84700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fafc4009f80 tx=0x7fafc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc4009320 con 0x7fafd00f5410 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafc4013ec0 con 0x7fafd00f5410 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd00fb9c0 con 0x7fafd00f5410 2023-11-16T01:13:37.352 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc401ece0 con 0x7fafd00f5410 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.349+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd00fbe50 con 0x7fafd00f5410 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.350+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafc401e420 con 0x7fafd00f5410 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.350+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd0059100 con 0x7fafd00f5410 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.350+0000 7fafcd7fa700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 0x7fafc0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.350+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafc4040d40 con 0x7fafd00f5410 2023-11-16T01:13:37.353 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.350+0000 7fafcffff700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 0x7fafc0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:37.358 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.357+0000 7fafcffff700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 0x7fafc0032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fafbc0096a0 tx=0x7fafbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:37.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.357+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafc401e780 con 0x7fafd00f5410 2023-11-16T01:13:37.565 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.564+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafd00fae60 con 0x7fafd00f5410 2023-11-16T01:13:37.566 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.565+0000 7fafcd7fa700 1 -- 172.21.15.134:0/4292646057 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafc401cb10 con 0x7fafd00f5410 2023-11-16T01:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 msgr2=0x7fafc0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 0x7fafc0032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fafbc0096a0 tx=0x7fafbc006b40).stop 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 msgr2=0x7fafd00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fafc4009f80 tx=0x7fafc400b040).stop 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 shutdown_connections 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fafc002ffb0 0x7fafc0032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 --2- 172.21.15.134:0/4292646057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafd00f5410 0x7fafd00fd3a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 >> 172.21.15.134:0/4292646057 conn(0x7fafd00f0b70 msgr2=0x7fafd005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 shutdown_connections 2023-11-16T01:13:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:37.566+0000 7fafd70e8700 1 -- 172.21.15.134:0/4292646057 wait complete. 2023-11-16T01:13:37.569 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:35.977505+0000 mgr.smithi133.xdyttl (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:13:36.431412+0000 mgr.smithi133.xdyttl (mgr.14162) 155 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi133 ... 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:09:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:09:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:10:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:11:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:12:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:12:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: main() 2023-11-16T01:13:37.880 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: r = ctx.func(ctx) 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: return func(ctx) 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-16T01:13:37.881 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: raise RuntimeError('uid/gid not found') 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: RuntimeError: uid/gid not found 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: yield (conn, connr) 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: code, '\n'.join(err))) 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi133 ... 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T01:13:37.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:09:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:09:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:10:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi133 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:11:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:12:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr time="2023-11-16T01:12:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-16T01:13:37.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: main() 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: r = ctx.func(ctx) 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: return func(ctx) 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-16T01:13:37.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: raise RuntimeError('uid/gid not found') 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: RuntimeError: uid/gid not found 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:13:36.436785+0000 mgr.smithi133.xdyttl (mgr.14162) 156 : cephadm [INF] Deploying daemon crash.smithi133 on smithi133 2023-11-16T01:13:37.885 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:37.566013+0000 mon.smithi133 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.134:0/4292646057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:38.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:38.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:39.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:37.922916+0000 mon.smithi133 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:13:39.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:38.029254+0000 mon.smithi133 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:13:39.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:38.031408+0000 mon.smithi133 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:13:39.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:38.031790+0000 mon.smithi133 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-16T01:13:39.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:38.037423+0000 mon.smithi133 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:13:40.074 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.072+0000 7f835debf700 1 -- 172.21.15.134:0/3537068375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 msgr2=0x7f83580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.072+0000 7f835debf700 1 --2- 172.21.15.134:0/3537068375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83580f66c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f834c009fb0 tx=0x7f834c009b30).stop 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.072+0000 7f835debf700 1 -- 172.21.15.134:0/3537068375 shutdown_connections 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.072+0000 7f835debf700 1 --2- 172.21.15.134:0/3537068375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83580f66c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.072+0000 7f835debf700 1 -- 172.21.15.134:0/3537068375 >> 172.21.15.134:0/3537068375 conn(0x7f83580f1aa0 msgr2=0x7f83580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.073+0000 7f835debf700 1 -- 172.21.15.134:0/3537068375 shutdown_connections 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.073+0000 7f835debf700 1 -- 172.21.15.134:0/3537068375 wait complete. 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.073+0000 7f835debf700 1 Processor -- start 2023-11-16T01:13:40.075 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.073+0000 7f835debf700 1 -- start start 2023-11-16T01:13:40.076 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835debf700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:40.076 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835debf700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83581069a0 con 0x7f83580f62a0 2023-11-16T01:13:40.076 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835cebd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:40.076 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835cebd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40756/0 (socket says 172.21.15.134:40756) 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835cebd700 1 -- 172.21.15.134:0/60242202 learned_addr learned my addr 172.21.15.134:0/60242202 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835cebd700 1 -- 172.21.15.134:0/60242202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c005040 con 0x7f83580f62a0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835cebd700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f834c013510 tx=0x7f834c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f834c009320 con 0x7f83580f62a0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8358106ba0 con 0x7f83580f62a0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f834c013f00 con 0x7f83580f62a0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f834c014dc0 con 0x7f83580f62a0 2023-11-16T01:13:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.074+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8358107030 con 0x7f83580f62a0 2023-11-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.075+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f834c0095f0 con 0x7f83580f62a0 2023-11-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.075+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8358045ad0 con 0x7f83580f62a0 2023-11-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.075+0000 7f83557fa700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 0x7f8348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.075+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f834c041000 con 0x7f83580f62a0 2023-11-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.076+0000 7f8357fff700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 0x7f8348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:40.079 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.076+0000 7f8357fff700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 0x7f8348032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f83440096a0 tx=0x7f8344006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:40.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.080+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f834c014420 con 0x7f83580f62a0 2023-11-16T01:13:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:37.977804+0000 mgr.smithi133.xdyttl (mgr.14162) 157 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:38.032210+0000 mgr.smithi133.xdyttl (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-16T01:13:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:13:38.040339+0000 mgr.smithi133.xdyttl (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi133 on smithi133 2023-11-16T01:13:40.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.288+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8358107ee0 con 0x7f83580f62a0 2023-11-16T01:13:40.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.289+0000 7f83557fa700 1 -- 172.21.15.134:0/60242202 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f834c010230 con 0x7f83580f62a0 2023-11-16T01:13:40.290 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:40.291 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 msgr2=0x7f8348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 0x7f8348032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f83440096a0 tx=0x7f8344006b40).stop 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 msgr2=0x7f83581064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f834c013510 tx=0x7f834c00b040).stop 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 shutdown_connections 2023-11-16T01:13:40.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f834802ff60 0x7f8348032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:40.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 --2- 172.21.15.134:0/60242202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83580f62a0 0x7f83581064b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:40.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 >> 172.21.15.134:0/60242202 conn(0x7f83580f1aa0 msgr2=0x7f83580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:40.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 shutdown_connections 2023-11-16T01:13:40.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:40.290+0000 7f835debf700 1 -- 172.21.15.134:0/60242202 wait complete. 2023-11-16T01:13:40.293 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:41.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:40.290437+0000 mon.smithi133 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.134:0/60242202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:41.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:41.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:42.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:39.978148+0000 mgr.smithi133.xdyttl (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- 172.21.15.134:0/2167519955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 msgr2=0x7f12880f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 --2- 172.21.15.134:0/2167519955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12880f9190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1278009fb0 tx=0x7f1278009b30).stop 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- 172.21.15.134:0/2167519955 shutdown_connections 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 --2- 172.21.15.134:0/2167519955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12880f9190 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- 172.21.15.134:0/2167519955 >> 172.21.15.134:0/2167519955 conn(0x7f12880f0b70 msgr2=0x7f12880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- 172.21.15.134:0/2167519955 shutdown_connections 2023-11-16T01:13:42.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- 172.21.15.134:0/2167519955 wait complete. 2023-11-16T01:13:42.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 Processor -- start 2023-11-16T01:13:42.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.818+0000 7f128d0cb700 1 -- start start 2023-11-16T01:13:42.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f128d0cb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:42.822 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f128d0cb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1288106bd0 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f1286d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f1286d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40768/0 (socket says 172.21.15.134:40768) 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f1286d9d700 1 -- 172.21.15.134:0/919782255 learned_addr learned my addr 172.21.15.134:0/919782255 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f1286d9d700 1 -- 172.21.15.134:0/919782255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278005040 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f1286d9d700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1278004dd0 tx=0x7f1278009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1278018070 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1288106dd0 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1278013410 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f127801c3c0 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.819+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1288107260 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.820+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f127801d420 con 0x7f12880601e0 2023-11-16T01:13:42.823 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.820+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1288045ad0 con 0x7f12880601e0 2023-11-16T01:13:42.824 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.820+0000 7f126f7fe700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 0x7f1270032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:42.824 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.820+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1278041310 con 0x7f12880601e0 2023-11-16T01:13:42.824 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.821+0000 7f128659c700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 0x7f1270032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:42.824 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.821+0000 7f128659c700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 0x7f1270032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f127c0096a0 tx=0x7f127c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:42.826 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:42.825+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f127801d820 con 0x7f12880601e0 2023-11-16T01:13:43.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.036+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12881080b0 con 0x7f12880601e0 2023-11-16T01:13:43.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.036+0000 7f126f7fe700 1 -- 172.21.15.134:0/919782255 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f127801b370 con 0x7f12880601e0 2023-11-16T01:13:43.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:43.038 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 msgr2=0x7f1270032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 0x7f1270032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f127c0096a0 tx=0x7f127c006b40).stop 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 msgr2=0x7f12881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1278004dd0 tx=0x7f1278009b30).stop 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 shutdown_connections 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f127002ff60 0x7f1270032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 --2- 172.21.15.134:0/919782255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f12880601e0 0x7f12881066e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 >> 172.21.15.134:0/919782255 conn(0x7f12880f0b70 msgr2=0x7f12880f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 shutdown_connections 2023-11-16T01:13:43.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:43.038+0000 7f128d0cb700 1 -- 172.21.15.134:0/919782255 wait complete. 2023-11-16T01:13:43.041 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:41.978507+0000 mgr.smithi133.xdyttl (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:43.037955+0000 mon.smithi133 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.134:0/919782255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:44.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:44.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:45.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- 172.21.15.134:0/3271224359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 msgr2=0x7fa7140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:45.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/3271224359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa7140f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fa708009fb0 tx=0x7fa708009b30).stop 2023-11-16T01:13:45.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- 172.21.15.134:0/3271224359 shutdown_connections 2023-11-16T01:13:45.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/3271224359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa7140f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:45.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- 172.21.15.134:0/3271224359 >> 172.21.15.134:0/3271224359 conn(0x7fa7140f0b70 msgr2=0x7fa7140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- 172.21.15.134:0/3271224359 shutdown_connections 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- 172.21.15.134:0/3271224359 wait complete. 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 Processor -- start 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.669+0000 7fa71bd2b700 1 -- start start 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa71bd2b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa71bd2b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa714105b20 con 0x7fa7140f5410 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa719ac7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa719ac7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40788/0 (socket says 172.21.15.134:40788) 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa719ac7700 1 -- 172.21.15.134:0/2084032851 learned_addr learned my addr 172.21.15.134:0/2084032851 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:45.672 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa719ac7700 1 -- 172.21.15.134:0/2084032851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708005040 con 0x7fa7140f5410 2023-11-16T01:13:45.673 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa719ac7700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa708013510 tx=0x7fa70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:45.673 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa708009320 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa708013f00 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa708014d90 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa714105d20 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.670+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7141061b0 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.671+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7080095f0 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.671+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa714045ad0 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.671+0000 7fa7067fc700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 0x7fa700032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.671+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa708041050 con 0x7fa7140f5410 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.671+0000 7fa7192c6700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 0x7fa700032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.672+0000 7fa7192c6700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 0x7fa700032470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa7100096a0 tx=0x7fa710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:45.677 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.676+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa708014420 con 0x7fa7140f5410 2023-11-16T01:13:45.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.878+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7140fae60 con 0x7fa7140f5410 2023-11-16T01:13:45.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.879+0000 7fa7067fc700 1 -- 172.21.15.134:0/2084032851 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa70801d7f0 con 0x7fa7140f5410 2023-11-16T01:13:45.880 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:45.881 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 msgr2=0x7fa700032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 0x7fa700032470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa7100096a0 tx=0x7fa710006b40).stop 2023-11-16T01:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 msgr2=0x7fa714105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa708013510 tx=0x7fa70800b040).stop 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 shutdown_connections 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa70002ffb0 0x7fa700032470 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 --2- 172.21.15.134:0/2084032851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7140f5410 0x7fa714105630 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 >> 172.21.15.134:0/2084032851 conn(0x7fa7140f0b70 msgr2=0x7fa7140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 shutdown_connections 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:45.881+0000 7fa71bd2b700 1 -- 172.21.15.134:0/2084032851 wait complete. 2023-11-16T01:13:45.883 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:43.978857+0000 mgr.smithi133.xdyttl (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:45.880421+0000 mon.smithi133 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.134:0/2084032851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:47.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:47.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:45.979090+0000 mgr.smithi133.xdyttl (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:48.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 -- 172.21.15.134:0/3454403808 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc4300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:48.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 --2- 172.21.15.134:0/3454403808 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fc420009fb0 tx=0x7fc420009b30).stop 2023-11-16T01:13:48.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 -- 172.21.15.134:0/3454403808 shutdown_connections 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 --2- 172.21.15.134:0/3454403808 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 -- 172.21.15.134:0/3454403808 >> 172.21.15.134:0/3454403808 conn(0x7fc4300f0b70 msgr2=0x7fc4300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 -- 172.21.15.134:0/3454403808 shutdown_connections 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.539+0000 7fc437427700 1 -- 172.21.15.134:0/3454403808 wait complete. 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc437427700 1 Processor -- start 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc437427700 1 -- start start 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc437427700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc437427700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430106c50 con 0x7fc4300f5410 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc4351c3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc4351c3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40812/0 (socket says 172.21.15.134:40812) 2023-11-16T01:13:48.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc4351c3700 1 -- 172.21.15.134:0/576225342 learned_addr learned my addr 172.21.15.134:0/576225342 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc4351c3700 1 -- 172.21.15.134:0/576225342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420005040 con 0x7fc4300f5410 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc4351c3700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc4200089f0 tx=0x7fc42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420009320 con 0x7fc4300f5410 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4200136c0 con 0x7fc4300f5410 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42001ecb0 con 0x7fc4300f5410 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.540+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430106e50 con 0x7fc4300f5410 2023-11-16T01:13:48.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.541+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4301072e0 con 0x7fc4300f5410 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc42001e420 con 0x7fc4300f5410 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430045ad0 con 0x7fc4300f5410 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc425ffb700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 0x7fc41c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc420040de0 con 0x7fc4300f5410 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc4349c2700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 0x7fc41c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.542+0000 7fc4349c2700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 0x7fc41c032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc42c0096a0 tx=0x7fc42c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:48.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.550+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc420045070 con 0x7fc4300f5410 2023-11-16T01:13:48.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.759+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc430108190 con 0x7fc4300f5410 2023-11-16T01:13:48.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.759+0000 7fc425ffb700 1 -- 172.21.15.134:0/576225342 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc42001e780 con 0x7fc4300f5410 2023-11-16T01:13:48.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:48.761 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 msgr2=0x7fc41c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 0x7fc41c032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc42c0096a0 tx=0x7fc42c006b40).stop 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc430106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fc4200089f0 tx=0x7fc42000b040).stop 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 shutdown_connections 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc41c02ffb0 0x7fc41c032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 --2- 172.21.15.134:0/576225342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc4300f5410 0x7fc430106760 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 >> 172.21.15.134:0/576225342 conn(0x7fc4300f0b70 msgr2=0x7fc43005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 shutdown_connections 2023-11-16T01:13:48.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:48.761+0000 7fc437427700 1 -- 172.21.15.134:0/576225342 wait complete. 2023-11-16T01:13:48.764 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:49.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:48.760807+0000 mon.smithi133 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.134:0/576225342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:50.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:50.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:47.979327+0000 mgr.smithi133.xdyttl (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:51.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:49.979538+0000 mgr.smithi133.xdyttl (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 -- 172.21.15.134:0/146927404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 msgr2=0x7fa6b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 --2- 172.21.15.134:0/146927404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b40f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fa6a4009fb0 tx=0x7fa6a4009b30).stop 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 -- 172.21.15.134:0/146927404 shutdown_connections 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 --2- 172.21.15.134:0/146927404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b40f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 -- 172.21.15.134:0/146927404 >> 172.21.15.134:0/146927404 conn(0x7fa6b40f0b70 msgr2=0x7fa6b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 -- 172.21.15.134:0/146927404 shutdown_connections 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.459+0000 7fa6bc384700 1 -- 172.21.15.134:0/146927404 wait complete. 2023-11-16T01:13:51.463 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6bc384700 1 Processor -- start 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6bc384700 1 -- start start 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6bc384700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6bc384700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4060240 con 0x7fa6b40f5410 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6ba120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6ba120700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56426/0 (socket says 172.21.15.134:56426) 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6ba120700 1 -- 172.21.15.134:0/1418784901 learned_addr learned my addr 172.21.15.134:0/1418784901 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6ba120700 1 -- 172.21.15.134:0/1418784901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4005040 con 0x7fa6b40f5410 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6ba120700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fa6a4014000 tx=0x7fa6a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a40095f0 con 0x7fa6b40f5410 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b4060440 con 0x7fa6b40f5410 2023-11-16T01:13:51.464 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.460+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6a4007e70 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4014ea0 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b4109ae0 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6a4025070 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b4045ad0 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6aaffd700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 0x7fa6a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.461+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6a40409f0 con 0x7fa6b40f5410 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.462+0000 7fa6b991f700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 0x7fa6a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:51.465 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.462+0000 7fa6b991f700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 0x7fa6a0032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa6b00096a0 tx=0x7fa6b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:51.468 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.466+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6a400f4e0 con 0x7fa6b40f5410 2023-11-16T01:13:51.673 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.671+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6b40fae60 con 0x7fa6b40f5410 2023-11-16T01:13:51.673 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.671+0000 7fa6aaffd700 1 -- 172.21.15.134:0/1418784901 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa6a400f300 con 0x7fa6b40f5410 2023-11-16T01:13:51.673 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:51.674 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:51.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 msgr2=0x7fa6a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 0x7fa6a0032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa6b00096a0 tx=0x7fa6b0006b40).stop 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 msgr2=0x7fa6b4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fa6a4014000 tx=0x7fa6a400b040).stop 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 shutdown_connections 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6a002ff60 0x7fa6a0032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 --2- 172.21.15.134:0/1418784901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6b40f5410 0x7fa6b4109430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 >> 172.21.15.134:0/1418784901 conn(0x7fa6b40f0b70 msgr2=0x7fa6b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 shutdown_connections 2023-11-16T01:13:51.675 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:51.673+0000 7fa6bc384700 1 -- 172.21.15.134:0/1418784901 wait complete. 2023-11-16T01:13:51.676 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:52.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:52 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:51.673081+0000 mon.smithi133 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.134:0/1418784901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:53.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:53.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:53.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:51.979789+0000 mgr.smithi133.xdyttl (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 -- 172.21.15.134:0/1204872664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 msgr2=0x7f4afc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 --2- 172.21.15.134:0/1204872664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f4af0009fb0 tx=0x7f4af0009b30).stop 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 -- 172.21.15.134:0/1204872664 shutdown_connections 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 --2- 172.21.15.134:0/1204872664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 -- 172.21.15.134:0/1204872664 >> 172.21.15.134:0/1204872664 conn(0x7f4afc0f0b70 msgr2=0x7f4afc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 -- 172.21.15.134:0/1204872664 shutdown_connections 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.248+0000 7f4b02e48700 1 -- 172.21.15.134:0/1204872664 wait complete. 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.249+0000 7f4b02e48700 1 Processor -- start 2023-11-16T01:13:54.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.249+0000 7f4b02e48700 1 -- start start 2023-11-16T01:13:54.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.249+0000 7f4b02e48700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:54.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.249+0000 7f4b02e48700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc0f9a00 con 0x7f4afc0f5410 2023-11-16T01:13:54.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.249+0000 7f4b00be4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:54.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b00be4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56440/0 (socket says 172.21.15.134:56440) 2023-11-16T01:13:54.252 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b00be4700 1 -- 172.21.15.134:0/2617550750 learned_addr learned my addr 172.21.15.134:0/2617550750 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:54.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b00be4700 1 -- 172.21.15.134:0/2617550750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0005040 con 0x7f4afc0f5410 2023-11-16T01:13:54.253 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b00be4700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4af0013510 tx=0x7f4af000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af0009320 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4af0013f00 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc0f9ba0 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af0014d90 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.250+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc0fa030 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.251+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4af0014420 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.251+0000 7f4af97fa700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 0x7f4aec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.251+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4af0041000 con 0x7f4afc0f5410 2023-11-16T01:13:54.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.251+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4afc059100 con 0x7f4afc0f5410 2023-11-16T01:13:54.255 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.251+0000 7f4afbfff700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 0x7f4aec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:54.255 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.252+0000 7f4afbfff700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 0x7f4aec032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4ae80096a0 tx=0x7f4ae8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:54.257 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.257+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4af00095f0 con 0x7f4afc0f5410 2023-11-16T01:13:54.467 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.465+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4afc0fa790 con 0x7f4afc0f5410 2023-11-16T01:13:54.467 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.466+0000 7f4af97fa700 1 -- 172.21.15.134:0/2617550750 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4af001db70 con 0x7f4afc0f5410 2023-11-16T01:13:54.467 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:54.468 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 msgr2=0x7f4aec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 0x7f4aec032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4ae80096a0 tx=0x7f4ae8006b40).stop 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 msgr2=0x7f4afc0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4af0013510 tx=0x7f4af000b040).stop 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 shutdown_connections 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4aec02ff60 0x7f4aec032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 --2- 172.21.15.134:0/2617550750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4afc0f5410 0x7f4afc0fb580 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 >> 172.21.15.134:0/2617550750 conn(0x7f4afc0f0b70 msgr2=0x7f4afc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 shutdown_connections 2023-11-16T01:13:54.469 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:54.467+0000 7f4b02e48700 1 -- 172.21.15.134:0/2617550750 wait complete. 2023-11-16T01:13:54.470 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:55.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:53.980074+0000 mgr.smithi133.xdyttl (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:55.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:54.305268+0000 mon.smithi133 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' 2023-11-16T01:13:55.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:13:54.306596+0000 mgr.smithi133.xdyttl (mgr.14162) 168 : cephadm [INF] Deploying daemon node-exporter.smithi133 on smithi133 2023-11-16T01:13:55.780 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:54.467194+0000 mon.smithi133 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.134:0/2617550750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:55.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:55.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:57.043 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 -- 172.21.15.134:0/2582907757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 msgr2=0x7f69c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:57.043 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 --2- 172.21.15.134:0/2582907757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c80f90f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f69bc009fb0 tx=0x7f69bc009b30).stop 2023-11-16T01:13:57.043 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 -- 172.21.15.134:0/2582907757 shutdown_connections 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 --2- 172.21.15.134:0/2582907757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c80f90f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 -- 172.21.15.134:0/2582907757 >> 172.21.15.134:0/2582907757 conn(0x7f69c80f0b70 msgr2=0x7f69c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 -- 172.21.15.134:0/2582907757 shutdown_connections 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.041+0000 7f69cd332700 1 -- 172.21.15.134:0/2582907757 wait complete. 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69cd332700 1 Processor -- start 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69cd332700 1 -- start start 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69cd332700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69cd332700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8106bd0 con 0x7f69c8060280 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69c6ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69c6ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:56454/0 (socket says 172.21.15.134:56454) 2023-11-16T01:13:57.044 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69c6ffd700 1 -- 172.21.15.134:0/1275510281 learned_addr learned my addr 172.21.15.134:0/1275510281 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.042+0000 7f69c6ffd700 1 -- 172.21.15.134:0/1275510281 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc005040 con 0x7f69c8060280 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69c6ffd700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f69bc009830 tx=0x7f69bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc009320 con 0x7f69c8060280 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c8106dd0 con 0x7f69c8060280 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69bc0136c0 con 0x7f69c8060280 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc01ee90 con 0x7f69c8060280 2023-11-16T01:13:57.045 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c8107260 con 0x7f69c8060280 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.043+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69bc01e420 con 0x7f69c8060280 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.044+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8045ad0 con 0x7f69c8060280 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.044+0000 7f69af7fe700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 0x7f69b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.044+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69bc040fc0 con 0x7f69c8060280 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.044+0000 7f69c67fc700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 0x7f69b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:57.046 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.044+0000 7f69c67fc700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 0x7f69b00323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f69b80096a0 tx=0x7f69b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:57.048 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.048+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69bc06d2b0 con 0x7f69c8060280 2023-11-16T01:13:57.254 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.254+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69c8108110 con 0x7f69c8060280 2023-11-16T01:13:57.255 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.255+0000 7f69af7fe700 1 -- 172.21.15.134:0/1275510281 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69bc0095f0 con 0x7f69c8060280 2023-11-16T01:13:57.256 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:13:57.256 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:13:57.257 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 msgr2=0x7f69b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 0x7f69b00323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f69b80096a0 tx=0x7f69b8006b40).stop 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 msgr2=0x7f69c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f69bc009830 tx=0x7f69bc00b040).stop 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 shutdown_connections 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f69b002ff10 0x7f69b00323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:57.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 --2- 172.21.15.134:0/1275510281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69c8060280 0x7f69c81066e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:57.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 >> 172.21.15.134:0/1275510281 conn(0x7f69c80f0b70 msgr2=0x7f69c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:57.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.256+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 shutdown_connections 2023-11-16T01:13:57.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:57.257+0000 7f69cd332700 1 -- 172.21.15.134:0/1275510281 wait complete. 2023-11-16T01:13:57.259 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:13:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:55.980588+0000 mgr.smithi133.xdyttl (mgr.14162) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:13:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:13:57.255101+0000 mon.smithi133 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.134:0/1275510281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:13:58.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:13:58.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 -- 172.21.15.134:0/946738249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 msgr2=0x7fceac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/946738249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac0f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fcea0009fb0 tx=0x7fcea0009b30).stop 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 -- 172.21.15.134:0/946738249 shutdown_connections 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/946738249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac0f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 -- 172.21.15.134:0/946738249 >> 172.21.15.134:0/946738249 conn(0x7fceac0f0b70 msgr2=0x7fceac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 -- 172.21.15.134:0/946738249 shutdown_connections 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.808+0000 7fceb3fd4700 1 -- 172.21.15.134:0/946738249 wait complete. 2023-11-16T01:13:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb3fd4700 1 Processor -- start 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb3fd4700 1 -- start start 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb3fd4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb3fd4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac105ac0 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb1d70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb1d70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34984/0 (socket says 172.21.15.134:34984) 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb1d70700 1 -- 172.21.15.134:0/591762653 learned_addr learned my addr 172.21.15.134:0/591762653 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb1d70700 1 -- 172.21.15.134:0/591762653 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0005040 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb1d70700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcea0013510 tx=0x7fcea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea0009320 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.809+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac105cc0 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac106150 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcea0013f00 con 0x7fceac0f5410 2023-11-16T01:13:59.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea0014d90 con 0x7fceac0f5410 2023-11-16T01:13:59.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcea00263a0 con 0x7fceac0f5410 2023-11-16T01:13:59.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fce9e7fc700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 0x7fce98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:13:59.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.811+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcea0040ff0 con 0x7fceac0f5410 2023-11-16T01:13:59.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.810+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac045ad0 con 0x7fceac0f5410 2023-11-16T01:13:59.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.811+0000 7fceb156f700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 0x7fce98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:13:59.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.811+0000 7fceb156f700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 0x7fce98032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcea80096a0 tx=0x7fcea8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:13:59.815 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:13:59.815+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcea0012920 con 0x7fceac0f5410 2023-11-16T01:14:00.024 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.023+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fceac106e90 con 0x7fceac0f5410 2023-11-16T01:14:00.024 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.024+0000 7fce9e7fc700 1 -- 172.21.15.134:0/591762653 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcea002b020 con 0x7fceac0f5410 2023-11-16T01:14:00.025 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:00.025 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:00.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 msgr2=0x7fce98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:00.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 0x7fce98032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcea80096a0 tx=0x7fcea8006b40).stop 2023-11-16T01:14:00.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 msgr2=0x7fceac1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:00.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcea0013510 tx=0x7fcea000b040).stop 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 shutdown_connections 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fce9802ffb0 0x7fce98032470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 --2- 172.21.15.134:0/591762653 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fceac0f5410 0x7fceac1055d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 >> 172.21.15.134:0/591762653 conn(0x7fceac0f0b70 msgr2=0x7fceac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 shutdown_connections 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:00.026+0000 7fceb3fd4700 1 -- 172.21.15.134:0/591762653 wait complete. 2023-11-16T01:14:00.027 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:00.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:13:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:57.980939+0000 mgr.smithi133.xdyttl (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:01.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:00.024570+0000 mon.smithi133 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.134:0/591762653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:01.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:01.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:13:59.981333+0000 mgr.smithi133.xdyttl (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 -- 172.21.15.134:0/1175696581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 msgr2=0x7f19ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 --2- 172.21.15.134:0/1175696581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac0f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f199c009fb0 tx=0x7f199c009b30).stop 2023-11-16T01:14:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 -- 172.21.15.134:0/1175696581 shutdown_connections 2023-11-16T01:14:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 --2- 172.21.15.134:0/1175696581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac0f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 -- 172.21.15.134:0/1175696581 >> 172.21.15.134:0/1175696581 conn(0x7f19ac0f0b70 msgr2=0x7f19ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 -- 172.21.15.134:0/1175696581 shutdown_connections 2023-11-16T01:14:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.666+0000 7f19b226d700 1 -- 172.21.15.134:0/1175696581 wait complete. 2023-11-16T01:14:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19b226d700 1 Processor -- start 2023-11-16T01:14:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19b226d700 1 -- start start 2023-11-16T01:14:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19b226d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19b226d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac106bd0 con 0x7f19ac0f5410 2023-11-16T01:14:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19abfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19abfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35008/0 (socket says 172.21.15.134:35008) 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.667+0000 7f19abfff700 1 -- 172.21.15.134:0/920772625 learned_addr learned my addr 172.21.15.134:0/920772625 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19abfff700 1 -- 172.21.15.134:0/920772625 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c005040 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19abfff700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f199c014000 tx=0x7f199c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199c0095f0 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac106dd0 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f199c007e70 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199c014ea0 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.668+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac107260 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.669+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f199c025070 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.669+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac045ad0 con 0x7f19ac0f5410 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.669+0000 7f19a8ff9700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 0x7f1998032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.669+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f199c040c90 con 0x7f19ac0f5410 2023-11-16T01:14:02.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.669+0000 7f19ab7fe700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 0x7f1998032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:02.671 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.670+0000 7f19ab7fe700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 0x7f1998032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f19940096a0 tx=0x7f1994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:02.674 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.674+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f199c00f710 con 0x7f19ac0f5410 2023-11-16T01:14:02.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.879+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19ac108110 con 0x7f19ac0f5410 2023-11-16T01:14:02.880 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.880+0000 7f19a8ff9700 1 -- 172.21.15.134:0/920772625 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f199c025300 con 0x7f19ac0f5410 2023-11-16T01:14:02.881 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:02.881 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 msgr2=0x7f1998032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 0x7f1998032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f19940096a0 tx=0x7f1994006b40).stop 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 msgr2=0x7f19ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f199c014000 tx=0x7f199c00b040).stop 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 shutdown_connections 2023-11-16T01:14:02.882 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f199802ff60 0x7f1998032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 --2- 172.21.15.134:0/920772625 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f19ac0f5410 0x7f19ac1066e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 >> 172.21.15.134:0/920772625 conn(0x7f19ac0f0b70 msgr2=0x7f19ac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 shutdown_connections 2023-11-16T01:14:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:02.882+0000 7f19b226d700 1 -- 172.21.15.134:0/920772625 wait complete. 2023-11-16T01:14:02.883 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:01.981730+0000 mgr.smithi133.xdyttl (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:02.880642+0000 mon.smithi133 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.134:0/920772625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:04.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:04.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- 172.21.15.134:0/4069094644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 msgr2=0x7fb80c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 --2- 172.21.15.134:0/4069094644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c0f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fb7fc009fb0 tx=0x7fb7fc009b30).stop 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- 172.21.15.134:0/4069094644 shutdown_connections 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 --2- 172.21.15.134:0/4069094644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c0f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- 172.21.15.134:0/4069094644 >> 172.21.15.134:0/4069094644 conn(0x7fb80c0f0b70 msgr2=0x7fb80c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- 172.21.15.134:0/4069094644 shutdown_connections 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- 172.21.15.134:0/4069094644 wait complete. 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 Processor -- start 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- start start 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:05.548 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.546+0000 7fb8134d9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c106c10 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb811275700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb811275700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35020/0 (socket says 172.21.15.134:35020) 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb811275700 1 -- 172.21.15.134:0/1072201519 learned_addr learned my addr 172.21.15.134:0/1072201519 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb811275700 1 -- 172.21.15.134:0/1072201519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc005040 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb811275700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb7fc013510 tx=0x7fb7fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7fc009320 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c106e10 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7fc013f00 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7fc014d90 con 0x7fb80c0f5410 2023-11-16T01:14:05.549 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.547+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c1072a0 con 0x7fb80c0f5410 2023-11-16T01:14:05.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.548+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7fc0263a0 con 0x7fb80c0f5410 2023-11-16T01:14:05.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.548+0000 7fb801ffb700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 0x7fb7f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:05.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.548+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7fc040fe0 con 0x7fb80c0f5410 2023-11-16T01:14:05.550 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.548+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c045ad0 con 0x7fb80c0f5410 2023-11-16T01:14:05.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.548+0000 7fb810a74700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 0x7fb7f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:05.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.549+0000 7fb810a74700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 0x7fb7f8032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fb8080096a0 tx=0x7fb808006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:05.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.553+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7fc012920 con 0x7fb80c0f5410 2023-11-16T01:14:05.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.756+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb80c107fe0 con 0x7fb80c0f5410 2023-11-16T01:14:05.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.757+0000 7fb801ffb700 1 -- 172.21.15.134:0/1072201519 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7fc00f340 con 0x7fb80c0f5410 2023-11-16T01:14:05.757 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:05.758 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:05.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 msgr2=0x7fb7f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:05.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 0x7fb7f8032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fb8080096a0 tx=0x7fb808006b40).stop 2023-11-16T01:14:05.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 msgr2=0x7fb80c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb7fc013510 tx=0x7fb7fc00b040).stop 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 shutdown_connections 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb7f802ff60 0x7fb7f8032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 --2- 172.21.15.134:0/1072201519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb80c0f5410 0x7fb80c106720 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 >> 172.21.15.134:0/1072201519 conn(0x7fb80c0f0b70 msgr2=0x7fb80c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 shutdown_connections 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:05.759+0000 7fb8134d9700 1 -- 172.21.15.134:0/1072201519 wait complete. 2023-11-16T01:14:05.760 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:03.982058+0000 mgr.smithi133.xdyttl (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:07.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:05.757448+0000 mon.smithi133 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.134:0/1072201519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:07.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:07.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:05.982395+0000 mgr.smithi133.xdyttl (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:08.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 -- 172.21.15.134:0/262935429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 msgr2=0x7fdaa40fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:08.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 --2- 172.21.15.134:0/262935429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa40fb550 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fdaa0009fb0 tx=0x7fdaa0009b30).stop 2023-11-16T01:14:08.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 -- 172.21.15.134:0/262935429 shutdown_connections 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 --2- 172.21.15.134:0/262935429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa40fb550 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 -- 172.21.15.134:0/262935429 >> 172.21.15.134:0/262935429 conn(0x7fdaa40f0b70 msgr2=0x7fdaa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 -- 172.21.15.134:0/262935429 shutdown_connections 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.383+0000 7fdaabffb700 1 -- 172.21.15.134:0/262935429 wait complete. 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaabffb700 1 Processor -- start 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaabffb700 1 -- start start 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaabffb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaabffb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa4108db0 con 0x7fdaa4060220 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaa9d97700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaa9d97700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35030/0 (socket says 172.21.15.134:35030) 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaa9d97700 1 -- 172.21.15.134:0/2222266848 learned_addr learned my addr 172.21.15.134:0/2222266848 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:08.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaa9d97700 1 -- 172.21.15.134:0/2222266848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0005040 con 0x7fdaa4060220 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaa9d97700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdaa0013510 tx=0x7fdaa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa0009320 con 0x7fdaa4060220 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaa0013f00 con 0x7fdaa4060220 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa4108fb0 con 0x7fdaa4060220 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.384+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa0014dc0 con 0x7fdaa4060220 2023-11-16T01:14:08.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.385+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa4109440 con 0x7fdaa4060220 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.385+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaa00095f0 con 0x7fdaa4060220 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.385+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa4045ad0 con 0x7fdaa4060220 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.385+0000 7fda9a7fc700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.385+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaa0041100 con 0x7fdaa4060220 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.386+0000 7fdaa9596700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.386+0000 7fdaa9596700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 0x7fda90032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fda940096a0 tx=0x7fda94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:08.390 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.390+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaa0014420 con 0x7fdaa4060220 2023-11-16T01:14:08.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.596+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaa410a2f0 con 0x7fdaa4060220 2023-11-16T01:14:08.597 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.596+0000 7fda9a7fc700 1 -- 172.21.15.134:0/2222266848 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdaa0010230 con 0x7fdaa4060220 2023-11-16T01:14:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:08.597 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:08.598 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 msgr2=0x7fda90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:08.598 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 0x7fda90032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fda940096a0 tx=0x7fda94006b40).stop 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 msgr2=0x7fdaa41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdaa0013510 tx=0x7fdaa000b040).stop 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 shutdown_connections 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fda9002ff60 0x7fda90032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 --2- 172.21.15.134:0/2222266848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdaa4060220 0x7fdaa41088c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 >> 172.21.15.134:0/2222266848 conn(0x7fdaa40f0b70 msgr2=0x7fdaa40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 shutdown_connections 2023-11-16T01:14:08.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:08.598+0000 7fdaabffb700 1 -- 172.21.15.134:0/2222266848 wait complete. 2023-11-16T01:14:08.600 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:09.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:08 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:08.596952+0000 mon.smithi133 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.134:0/2222266848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:09.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:09.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:07.982682+0000 mgr.smithi133.xdyttl (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 -- 172.21.15.134:0/3175460388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 msgr2=0x7f9dc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 --2- 172.21.15.134:0/3175460388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc00f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f9db0009fb0 tx=0x7f9db0009b30).stop 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 -- 172.21.15.134:0/3175460388 shutdown_connections 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 --2- 172.21.15.134:0/3175460388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc00f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 -- 172.21.15.134:0/3175460388 >> 172.21.15.134:0/3175460388 conn(0x7f9dc00f0b70 msgr2=0x7f9dc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:11.160 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 -- 172.21.15.134:0/3175460388 shutdown_connections 2023-11-16T01:14:11.161 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.158+0000 7f9dc581c700 1 -- 172.21.15.134:0/3175460388 wait complete. 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.159+0000 7f9dc581c700 1 Processor -- start 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.159+0000 7f9dc581c700 1 -- start start 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.159+0000 7f9dc581c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.159+0000 7f9dc581c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0060240 con 0x7f9dc00f5410 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dbeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dbeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37426/0 (socket says 172.21.15.134:37426) 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dbeffd700 1 -- 172.21.15.134:0/2321632932 learned_addr learned my addr 172.21.15.134:0/2321632932 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dbeffd700 1 -- 172.21.15.134:0/2321632932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db0005040 con 0x7f9dc00f5410 2023-11-16T01:14:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dbeffd700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9db0013510 tx=0x7f9db000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db0009320 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9db0013f00 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc0060440 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db0014d90 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.160+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc0109ab0 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.161+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9db0014420 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.161+0000 7f9da77fe700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 0x7f9da8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.161+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc0045ad0 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.161+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9db0041000 con 0x7f9dc00f5410 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.161+0000 7f9dbe7fc700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 0x7f9da8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.162+0000 7f9dbe7fc700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 0x7f9da8032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9db40096a0 tx=0x7f9db4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:11.168 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.166+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9db00095f0 con 0x7f9dc00f5410 2023-11-16T01:14:11.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.371+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dc00faf50 con 0x7f9dc00f5410 2023-11-16T01:14:11.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.372+0000 7f9da77fe700 1 -- 172.21.15.134:0/2321632932 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9db002d090 con 0x7f9dc00f5410 2023-11-16T01:14:11.373 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:11.374 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.373+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 msgr2=0x7f9da8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.373+0000 7f9dc581c700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 0x7f9da8032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9db40096a0 tx=0x7f9db4006b40).stop 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.373+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 msgr2=0x7f9dc0109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.373+0000 7f9dc581c700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9db0013510 tx=0x7f9db000b040).stop 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 shutdown_connections 2023-11-16T01:14:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9da802ff60 0x7f9da8032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:11.376 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 --2- 172.21.15.134:0/2321632932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dc00f5410 0x7f9dc0109400 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:11.376 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 >> 172.21.15.134:0/2321632932 conn(0x7f9dc00f0b70 msgr2=0x7f9dc00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:11.376 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 shutdown_connections 2023-11-16T01:14:11.376 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:11.374+0000 7f9dc581c700 1 -- 172.21.15.134:0/2321632932 wait complete. 2023-11-16T01:14:11.376 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:09.983043+0000 mgr.smithi133.xdyttl (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:11.373480+0000 mon.smithi133 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.134:0/2321632932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:12.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:12.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 -- 172.21.15.134:0/1030967436 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 msgr2=0x7fe7100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 --2- 172.21.15.134:0/1030967436 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe7100f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe704009fb0 tx=0x7fe704009b30).stop 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 -- 172.21.15.134:0/1030967436 shutdown_connections 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 --2- 172.21.15.134:0/1030967436 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe7100f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 -- 172.21.15.134:0/1030967436 >> 172.21.15.134:0/1030967436 conn(0x7fe7100f0b70 msgr2=0x7fe7100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 -- 172.21.15.134:0/1030967436 shutdown_connections 2023-11-16T01:14:13.948 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.946+0000 7fe71710f700 1 -- 172.21.15.134:0/1030967436 wait complete. 2023-11-16T01:14:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe71710f700 1 Processor -- start 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe71710f700 1 -- start start 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe71710f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe71710f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe710105b20 con 0x7fe7100f5410 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe714eab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe714eab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37434/0 (socket says 172.21.15.134:37434) 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe714eab700 1 -- 172.21.15.134:0/300984128 learned_addr learned my addr 172.21.15.134:0/300984128 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.947+0000 7fe714eab700 1 -- 172.21.15.134:0/300984128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704005040 con 0x7fe7100f5410 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe714eab700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe704013510 tx=0x7fe70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe704009320 con 0x7fe7100f5410 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe710105d20 con 0x7fe7100f5410 2023-11-16T01:14:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe704013f00 con 0x7fe7100f5410 2023-11-16T01:14:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe704014d90 con 0x7fe7100f5410 2023-11-16T01:14:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.948+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7101061b0 con 0x7fe7100f5410 2023-11-16T01:14:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7040095f0 con 0x7fe7100f5410 2023-11-16T01:14:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe710045ad0 con 0x7fe7100f5410 2023-11-16T01:14:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe70d7fa700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7040410c0 con 0x7fe7100f5410 2023-11-16T01:14:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe70ffff700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.949+0000 7fe70ffff700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 0x7fe700032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:13.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:13.953+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7040268c0 con 0x7fe7100f5410 2023-11-16T01:14:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:11.983373+0000 mgr.smithi133.xdyttl (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:14.163 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.161+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7100fae60 con 0x7fe7100f5410 2023-11-16T01:14:14.164 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.162+0000 7fe70d7fa700 1 -- 172.21.15.134:0/300984128 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe704010230 con 0x7fe7100f5410 2023-11-16T01:14:14.164 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:14.164 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.163+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 msgr2=0x7fe700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.163+0000 7fe71710f700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 0x7fe700032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).stop 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.163+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 msgr2=0x7fe710105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.163+0000 7fe71710f700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe704013510 tx=0x7fe70400b040).stop 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 shutdown_connections 2023-11-16T01:14:14.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:14.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 --2- 172.21.15.134:0/300984128 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7100f5410 0x7fe710105630 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:14.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 >> 172.21.15.134:0/300984128 conn(0x7fe7100f0b70 msgr2=0x7fe7100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:14.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 shutdown_connections 2023-11-16T01:14:14.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:14.164+0000 7fe71710f700 1 -- 172.21.15.134:0/300984128 wait complete. 2023-11-16T01:14:14.166 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:15.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:14 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:14.163561+0000 mon.smithi133 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.134:0/300984128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:15.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:15.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:13.983687+0000 mgr.smithi133.xdyttl (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:16.719 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 -- 172.21.15.134:0/335489716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 msgr2=0x7fbb540fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 --2- 172.21.15.134:0/335489716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb540fb570 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fbb3c009fb0 tx=0x7fbb3c009b30).stop 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 -- 172.21.15.134:0/335489716 shutdown_connections 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 --2- 172.21.15.134:0/335489716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb540fb570 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 -- 172.21.15.134:0/335489716 >> 172.21.15.134:0/335489716 conn(0x7fbb540f0b70 msgr2=0x7fbb540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 -- 172.21.15.134:0/335489716 shutdown_connections 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 -- 172.21.15.134:0/335489716 wait complete. 2023-11-16T01:14:16.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.716+0000 7fbb59c96700 1 Processor -- start 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb59c96700 1 -- start start 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb59c96700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb59c96700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54108dc0 con 0x7fbb54060200 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb537fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb537fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:37454/0 (socket says 172.21.15.134:37454) 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb537fe700 1 -- 172.21.15.134:0/2948120708 learned_addr learned my addr 172.21.15.134:0/2948120708 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.717+0000 7fbb537fe700 1 -- 172.21.15.134:0/2948120708 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c005040 con 0x7fbb54060200 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb537fe700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fbb3c013510 tx=0x7fbb3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:16.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb3c009320 con 0x7fbb54060200 2023-11-16T01:14:16.722 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb54108fc0 con 0x7fbb54060200 2023-11-16T01:14:16.722 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb3c013f00 con 0x7fbb54060200 2023-11-16T01:14:16.722 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb3c014dc0 con 0x7fbb54060200 2023-11-16T01:14:16.722 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.718+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb54109450 con 0x7fbb54060200 2023-11-16T01:14:16.723 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb3c014420 con 0x7fbb54060200 2023-11-16T01:14:16.723 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb54045ad0 con 0x7fbb54060200 2023-11-16T01:14:16.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb4bfff700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 0x7fbb40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:16.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb3c041050 con 0x7fbb54060200 2023-11-16T01:14:16.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb52ffd700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 0x7fbb40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:16.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.719+0000 7fbb52ffd700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 0x7fbb40032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fbb440096a0 tx=0x7fbb44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:16.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.723+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb3c0095f0 con 0x7fbb54060200 2023-11-16T01:14:16.933 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.931+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb5410a300 con 0x7fbb54060200 2023-11-16T01:14:16.933 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.932+0000 7fbb4bfff700 1 -- 172.21.15.134:0/2948120708 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb3c01cb10 con 0x7fbb54060200 2023-11-16T01:14:16.934 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:16.934 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.933+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 msgr2=0x7fbb40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.933+0000 7fbb59c96700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 0x7fbb40032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fbb440096a0 tx=0x7fbb44006b40).stop 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.933+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 msgr2=0x7fbb541088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.933+0000 7fbb59c96700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fbb3c013510 tx=0x7fbb3c00b040).stop 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 shutdown_connections 2023-11-16T01:14:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fbb4002ff60 0x7fbb40032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:16.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 --2- 172.21.15.134:0/2948120708 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb54060200 0x7fbb541088d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:16.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 >> 172.21.15.134:0/2948120708 conn(0x7fbb540f0b70 msgr2=0x7fbb540f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:16.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 shutdown_connections 2023-11-16T01:14:16.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:16.934+0000 7fbb59c96700 1 -- 172.21.15.134:0/2948120708 wait complete. 2023-11-16T01:14:16.936 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:15.984021+0000 mgr.smithi133.xdyttl (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:16.933591+0000 mon.smithi133 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.134:0/2948120708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:18.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:18.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:19.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 -- 172.21.15.134:0/1459765927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 msgr2=0x7f72e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:19.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 --2- 172.21.15.134:0/1459765927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e40f55c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f72d8009fb0 tx=0x7f72d8009b30).stop 2023-11-16T01:14:19.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 -- 172.21.15.134:0/1459765927 shutdown_connections 2023-11-16T01:14:19.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 --2- 172.21.15.134:0/1459765927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e40f55c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:19.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 -- 172.21.15.134:0/1459765927 >> 172.21.15.134:0/1459765927 conn(0x7f72e40f09a0 msgr2=0x7f72e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:19.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.469+0000 7f72e89a4700 1 -- 172.21.15.134:0/1459765927 shutdown_connections 2023-11-16T01:14:19.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e89a4700 1 -- 172.21.15.134:0/1459765927 wait complete. 2023-11-16T01:14:19.473 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e89a4700 1 Processor -- start 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e89a4700 1 -- start start 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e89a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e89a4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e4060220 con 0x7f72e40f51a0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e2d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e2d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54748/0 (socket says 172.21.15.134:54748) 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.470+0000 7f72e2d9d700 1 -- 172.21.15.134:0/2815198809 learned_addr learned my addr 172.21.15.134:0/2815198809 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72e2d9d700 1 -- 172.21.15.134:0/2815198809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8005040 con 0x7f72e40f51a0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72e2d9d700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f72d8009830 tx=0x7f72d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d8009320 con 0x7f72e40f51a0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e4060420 con 0x7f72e40f51a0 2023-11-16T01:14:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72d80136c0 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d801ee90 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.471+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e4109880 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.472+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72d801e420 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.472+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e4059100 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.472+0000 7f72cb7fe700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 0x7f72cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.472+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72d8040fe0 con 0x7f72e40f51a0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.472+0000 7f72e259c700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 0x7f72cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:19.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.473+0000 7f72e259c700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 0x7f72cc0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f72d40096a0 tx=0x7f72d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:19.478 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.477+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72d806d2b0 con 0x7f72e40f51a0 2023-11-16T01:14:19.687 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.686+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72e4045ad0 con 0x7f72e40f51a0 2023-11-16T01:14:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.686+0000 7f72cb7fe700 1 -- 172.21.15.134:0/2815198809 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f72d801c790 con 0x7f72e40f51a0 2023-11-16T01:14:19.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:19.688 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:19.689 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 msgr2=0x7f72cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 0x7f72cc0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f72d40096a0 tx=0x7f72d4006b40).stop 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 msgr2=0x7f72e41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f72d8009830 tx=0x7f72d800b040).stop 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 shutdown_connections 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f72cc02ff10 0x7f72cc0323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 --2- 172.21.15.134:0/2815198809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f72e40f51a0 0x7f72e41091d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 >> 172.21.15.134:0/2815198809 conn(0x7f72e40f09a0 msgr2=0x7f72e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 shutdown_connections 2023-11-16T01:14:19.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:19.688+0000 7f72e89a4700 1 -- 172.21.15.134:0/2815198809 wait complete. 2023-11-16T01:14:19.691 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:17.984395+0000 mgr.smithi133.xdyttl (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:19.687884+0000 mon.smithi133 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.134:0/2815198809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:21.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:21.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:22.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:19.984668+0000 mgr.smithi133.xdyttl (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f7001ffb700 1 -- 172.21.15.134:0/685083950 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ff40138a0 con 0x7f70040f47c0 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 -- 172.21.15.134:0/685083950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 msgr2=0x7f70040f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 --2- 172.21.15.134:0/685083950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f70040f4be0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f6ff4009fb0 tx=0x7f6ff4009b30).stop 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 -- 172.21.15.134:0/685083950 shutdown_connections 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 --2- 172.21.15.134:0/685083950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f70040f4be0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 -- 172.21.15.134:0/685083950 >> 172.21.15.134:0/685083950 conn(0x7f70040f02e0 msgr2=0x7f70040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:22.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 -- 172.21.15.134:0/685083950 shutdown_connections 2023-11-16T01:14:22.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.173+0000 7f70093ce700 1 -- 172.21.15.134:0/685083950 wait complete. 2023-11-16T01:14:22.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f70093ce700 1 Processor -- start 2023-11-16T01:14:22.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f70093ce700 1 -- start start 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f70093ce700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f70093ce700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004108df0 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f7002ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f7002ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54758/0 (socket says 172.21.15.134:54758) 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.174+0000 7f7002ffd700 1 -- 172.21.15.134:0/4230431419 learned_addr learned my addr 172.21.15.134:0/4230431419 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f7002ffd700 1 -- 172.21.15.134:0/4230431419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4005040 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f7002ffd700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f6ff4013510 tx=0x7f6ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ff4009320 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7004108ff0 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ff4013f00 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ff4014d90 con 0x7f70040f47c0 2023-11-16T01:14:22.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.175+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7004109480 con 0x7f70040f47c0 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.176+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ff40095f0 con 0x7f70040f47c0 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.176+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004045ad0 con 0x7f70040f47c0 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.176+0000 7f6feb7fe700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 0x7f6fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.176+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ff4041080 con 0x7f70040f47c0 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.176+0000 7f70027fc700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 0x7f6fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:22.179 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.177+0000 7f70027fc700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 0x7f6fec032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f6ff80096a0 tx=0x7f6ff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:22.182 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.181+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ff4014420 con 0x7f70040f47c0 2023-11-16T01:14:22.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.387+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70040f6b20 con 0x7f70040f47c0 2023-11-16T01:14:22.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.389+0000 7f6feb7fe700 1 -- 172.21.15.134:0/4230431419 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ff4010230 con 0x7f70040f47c0 2023-11-16T01:14:22.390 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:22.390 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:22.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.389+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 msgr2=0x7f6fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:22.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.389+0000 7f70093ce700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 0x7f6fec032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f6ff80096a0 tx=0x7f6ff8006b40).stop 2023-11-16T01:14:22.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 msgr2=0x7f7004108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:22.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f6ff4013510 tx=0x7f6ff400b040).stop 2023-11-16T01:14:22.391 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 shutdown_connections 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6fec02ff60 0x7f6fec032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 --2- 172.21.15.134:0/4230431419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70040f47c0 0x7f7004108900 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 >> 172.21.15.134:0/4230431419 conn(0x7f70040f02e0 msgr2=0x7f70040f10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 shutdown_connections 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:22.390+0000 7f70093ce700 1 -- 172.21.15.134:0/4230431419 wait complete. 2023-11-16T01:14:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:23.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:22.389444+0000 mon.smithi133 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.134:0/4230431419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:23.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:23.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:21.984999+0000 mgr.smithi133.xdyttl (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:24.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 -- 172.21.15.134:0/2286843959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 msgr2=0x7f052c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:24.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 --2- 172.21.15.134:0/2286843959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c0f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f051c009fb0 tx=0x7f051c009b30).stop 2023-11-16T01:14:24.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 -- 172.21.15.134:0/2286843959 shutdown_connections 2023-11-16T01:14:24.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 --2- 172.21.15.134:0/2286843959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c0f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 -- 172.21.15.134:0/2286843959 >> 172.21.15.134:0/2286843959 conn(0x7f052c0f0b70 msgr2=0x7f052c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 -- 172.21.15.134:0/2286843959 shutdown_connections 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.972+0000 7f0531619700 1 -- 172.21.15.134:0/2286843959 wait complete. 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f0531619700 1 Processor -- start 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f0531619700 1 -- start start 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f0531619700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f0531619700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c060240 con 0x7f052c0f5410 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f052affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f052affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54772/0 (socket says 172.21.15.134:54772) 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f052affd700 1 -- 172.21.15.134:0/2365379377 learned_addr learned my addr 172.21.15.134:0/2365379377 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.973+0000 7f052affd700 1 -- 172.21.15.134:0/2365379377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c005040 con 0x7f052c0f5410 2023-11-16T01:14:24.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f052affd700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f051c013510 tx=0x7f051c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c009320 con 0x7f052c0f5410 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c060440 con 0x7f052c0f5410 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f051c013f00 con 0x7f052c0f5410 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c014d90 con 0x7f052c0f5410 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.974+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c109ae0 con 0x7f052c0f5410 2023-11-16T01:14:24.977 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.975+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f051c014420 con 0x7f052c0f5410 2023-11-16T01:14:24.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.975+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c045ad0 con 0x7f052c0f5410 2023-11-16T01:14:24.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.975+0000 7f05137fe700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 0x7f0514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:24.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.975+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f051c041000 con 0x7f052c0f5410 2023-11-16T01:14:24.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.975+0000 7f052a7fc700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 0x7f0514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:24.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.976+0000 7f052a7fc700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 0x7f0514032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f05200096a0 tx=0x7f0520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:24.981 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:24.980+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f051c0095f0 con 0x7f052c0f5410 2023-11-16T01:14:25.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.183+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f052c0fae60 con 0x7f052c0f5410 2023-11-16T01:14:25.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.184+0000 7f05137fe700 1 -- 172.21.15.134:0/2365379377 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f051c01db70 con 0x7f052c0f5410 2023-11-16T01:14:25.185 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:25.186 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:25.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 msgr2=0x7f0514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:25.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 0x7f0514032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f05200096a0 tx=0x7f0520006b40).stop 2023-11-16T01:14:25.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 msgr2=0x7f052c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f051c013510 tx=0x7f051c00b040).stop 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 shutdown_connections 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f051402ff60 0x7f0514032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 --2- 172.21.15.134:0/2365379377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f052c0f5410 0x7f052c109430 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 >> 172.21.15.134:0/2365379377 conn(0x7f052c0f0b70 msgr2=0x7f052c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.185+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 shutdown_connections 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:25.186+0000 7f0531619700 1 -- 172.21.15.134:0/2365379377 wait complete. 2023-11-16T01:14:25.188 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:23.985376+0000 mgr.smithi133.xdyttl (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:25.185520+0000 mon.smithi133 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.134:0/2365379377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:26.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:26.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- 172.21.15.134:0/837352605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 msgr2=0x7f03d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/837352605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d40f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f03d0009fb0 tx=0x7f03d0009b30).stop 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- 172.21.15.134:0/837352605 shutdown_connections 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/837352605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d40f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- 172.21.15.134:0/837352605 >> 172.21.15.134:0/837352605 conn(0x7f03d40f0b70 msgr2=0x7f03d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:27.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- 172.21.15.134:0/837352605 shutdown_connections 2023-11-16T01:14:27.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- 172.21.15.134:0/837352605 wait complete. 2023-11-16T01:14:27.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 Processor -- start 2023-11-16T01:14:27.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.754+0000 7f03dc7e9700 1 -- start start 2023-11-16T01:14:27.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03dc7e9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:27.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03dc7e9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4060240 con 0x7f03d40f5410 2023-11-16T01:14:27.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03da585700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:27.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03da585700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:54798/0 (socket says 172.21.15.134:54798) 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03da585700 1 -- 172.21.15.134:0/2916805630 learned_addr learned my addr 172.21.15.134:0/2916805630 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03da585700 1 -- 172.21.15.134:0/2916805630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0005040 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.755+0000 7f03da585700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f03d0013510 tx=0x7f03d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d0009320 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d4060440 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03d0013f00 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d0014d90 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d4109a80 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03d00095f0 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.756+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d4045ad0 con 0x7f03d40f5410 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.757+0000 7f03caffd700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 0x7f03c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:27.759 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.757+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03d0041000 con 0x7f03d40f5410 2023-11-16T01:14:27.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.757+0000 7f03d9d84700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 0x7f03c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:27.760 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.757+0000 7f03d9d84700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 0x7f03c0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f03c40096a0 tx=0x7f03c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:27.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.761+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03d001d7f0 con 0x7f03d40f5410 2023-11-16T01:14:27.972 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.971+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03d410a2a0 con 0x7f03d40f5410 2023-11-16T01:14:27.973 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.971+0000 7f03caffd700 1 -- 172.21.15.134:0/2916805630 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f03d0010230 con 0x7f03d40f5410 2023-11-16T01:14:27.973 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:27.973 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:27.974 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 msgr2=0x7f03c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:27.974 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 0x7f03c0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f03c40096a0 tx=0x7f03c4006b40).stop 2023-11-16T01:14:27.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 msgr2=0x7f03d41093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:27.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f03d0013510 tx=0x7f03d000b040).stop 2023-11-16T01:14:27.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 shutdown_connections 2023-11-16T01:14:27.975 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f03c002ff60 0x7f03c0032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:27.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 --2- 172.21.15.134:0/2916805630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f03d40f5410 0x7f03d41093d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:27.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 >> 172.21.15.134:0/2916805630 conn(0x7f03d40f0b70 msgr2=0x7f03d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:27.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 shutdown_connections 2023-11-16T01:14:27.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:27.973+0000 7f03dc7e9700 1 -- 172.21.15.134:0/2916805630 wait complete. 2023-11-16T01:14:27.979 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:25.985711+0000 mgr.smithi133.xdyttl (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:29.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:28 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:27.972890+0000 mon.smithi133 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.134:0/2916805630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:29.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:29.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:30.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:27.986050+0000 mgr.smithi133.xdyttl (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 -- 172.21.15.134:0/863110312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 msgr2=0x7f9ac00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 --2- 172.21.15.134:0/863110312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac00fb570 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f9aa8009fb0 tx=0x7f9aa8009b30).stop 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 -- 172.21.15.134:0/863110312 shutdown_connections 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 --2- 172.21.15.134:0/863110312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac00fb570 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 -- 172.21.15.134:0/863110312 >> 172.21.15.134:0/863110312 conn(0x7f9ac00f0b70 msgr2=0x7f9ac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 -- 172.21.15.134:0/863110312 shutdown_connections 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 -- 172.21.15.134:0/863110312 wait complete. 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.514+0000 7f9ac6035700 1 Processor -- start 2023-11-16T01:14:30.516 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9ac6035700 1 -- start start 2023-11-16T01:14:30.518 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9ac6035700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:30.518 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9ac6035700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0108dc0 con 0x7f9ac0060200 2023-11-16T01:14:30.518 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9abf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9abf7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35386/0 (socket says 172.21.15.134:35386) 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9abf7fe700 1 -- 172.21.15.134:0/1311229997 learned_addr learned my addr 172.21.15.134:0/1311229997 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.515+0000 7f9abf7fe700 1 -- 172.21.15.134:0/1311229997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8005040 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9abf7fe700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9aa8013510 tx=0x7f9aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa8009320 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aa8013f00 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0108fc0 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa8014dc0 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.516+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0109450 con 0x7f9ac0060200 2023-11-16T01:14:30.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9aa801d3d0 con 0x7f9ac0060200 2023-11-16T01:14:30.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9ab7fff700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 0x7f9aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:30.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9aa8041120 con 0x7f9ac0060200 2023-11-16T01:14:30.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac0059100 con 0x7f9ac0060200 2023-11-16T01:14:30.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9abeffd700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 0x7f9aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:30.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.517+0000 7f9abeffd700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 0x7f9aac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9ab00096a0 tx=0x7f9ab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:30.523 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.522+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9aa8012900 con 0x7f9ac0060200 2023-11-16T01:14:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.729+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ac00fe150 con 0x7f9ac0060200 2023-11-16T01:14:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.730+0000 7f9ab7fff700 1 -- 172.21.15.134:0/1311229997 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9aa802b020 con 0x7f9ac0060200 2023-11-16T01:14:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:30.732 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.731+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 msgr2=0x7f9aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.731+0000 7f9ac6035700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 0x7f9aac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9ab00096a0 tx=0x7f9ab0006b40).stop 2023-11-16T01:14:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.731+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 msgr2=0x7f9ac01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.731+0000 7f9ac6035700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9aa8013510 tx=0x7f9aa800b040).stop 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 shutdown_connections 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9aac02ff60 0x7f9aac032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 --2- 172.21.15.134:0/1311229997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ac0060200 0x7f9ac01088d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 >> 172.21.15.134:0/1311229997 conn(0x7f9ac00f0b70 msgr2=0x7f9ac00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 shutdown_connections 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:30.732+0000 7f9ac6035700 1 -- 172.21.15.134:0/1311229997 wait complete. 2023-11-16T01:14:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:30.076069+0000 mon.smithi133 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:14:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:30.077059+0000 mon.smithi133 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:14:32.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:29.986405+0000 mgr.smithi133.xdyttl (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:32.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:30.731546+0000 mon.smithi133 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.134:0/1311229997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:32.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:32.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 -- 172.21.15.134:0/1245395154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 msgr2=0x7f436c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 --2- 172.21.15.134:0/1245395154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c0f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f4364009960 tx=0x7f4364004d10).stop 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 -- 172.21.15.134:0/1245395154 shutdown_connections 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 --2- 172.21.15.134:0/1245395154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c0f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 -- 172.21.15.134:0/1245395154 >> 172.21.15.134:0/1245395154 conn(0x7f436c0f0b70 msgr2=0x7f436c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 -- 172.21.15.134:0/1245395154 shutdown_connections 2023-11-16T01:14:33.316 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.314+0000 7f4373b64700 1 -- 172.21.15.134:0/1245395154 wait complete. 2023-11-16T01:14:33.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4373b64700 1 Processor -- start 2023-11-16T01:14:33.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4373b64700 1 -- start start 2023-11-16T01:14:33.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4373b64700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:33.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4373b64700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c060220 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4371900700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4371900700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35406/0 (socket says 172.21.15.134:35406) 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4371900700 1 -- 172.21.15.134:0/2811197333 learned_addr learned my addr 172.21.15.134:0/2811197333 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4371900700 1 -- 172.21.15.134:0/2811197333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364009640 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.315+0000 7f4371900700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f4364004b70 tx=0x7f4364004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4364018070 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f436400b870 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c060420 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f436401c560 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c109aa0 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.316+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4364013630 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.317+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c059100 con 0x7f436c0f5410 2023-11-16T01:14:33.319 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.317+0000 7f43627fc700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 0x7f4358032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:33.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.317+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4364041250 con 0x7f436c0f5410 2023-11-16T01:14:33.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.317+0000 7f43710ff700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 0x7f4358032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:33.320 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.317+0000 7f43710ff700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 0x7f4358032470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f435c0096a0 tx=0x7f435c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:33.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.321+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4364013a30 con 0x7f436c0f5410 2023-11-16T01:14:33.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.525+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f436c045ad0 con 0x7f436c0f5410 2023-11-16T01:14:33.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.526+0000 7f43627fc700 1 -- 172.21.15.134:0/2811197333 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4364004e10 con 0x7f436c0f5410 2023-11-16T01:14:33.528 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:33.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 msgr2=0x7f4358032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:33.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 0x7f4358032470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f435c0096a0 tx=0x7f435c006b40).stop 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 msgr2=0x7f436c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f4364004b70 tx=0x7f4364004d10).stop 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 shutdown_connections 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f435802ffb0 0x7f4358032470 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 --2- 172.21.15.134:0/2811197333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f436c0f5410 0x7f436c1093f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 >> 172.21.15.134:0/2811197333 conn(0x7f436c0f0b70 msgr2=0x7f436c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 shutdown_connections 2023-11-16T01:14:33.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:33.528+0000 7f4373b64700 1 -- 172.21.15.134:0/2811197333 wait complete. 2023-11-16T01:14:33.531 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:31.986768+0000 mgr.smithi133.xdyttl (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:33.527571+0000 mon.smithi133 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.134:0/2811197333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:34.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:34.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:33.987080+0000 mgr.smithi133.xdyttl (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- 172.21.15.134:0/1420250154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 msgr2=0x7f6a400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 --2- 172.21.15.134:0/1420250154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a400f55c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f6a3c009fb0 tx=0x7f6a3c009b30).stop 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- 172.21.15.134:0/1420250154 shutdown_connections 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 --2- 172.21.15.134:0/1420250154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a400f55c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- 172.21.15.134:0/1420250154 >> 172.21.15.134:0/1420250154 conn(0x7f6a400f09a0 msgr2=0x7f6a400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- 172.21.15.134:0/1420250154 shutdown_connections 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- 172.21.15.134:0/1420250154 wait complete. 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 Processor -- start 2023-11-16T01:14:36.081 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- start start 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.078+0000 7f6a469a4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a40106ff0 con 0x7f6a400f51a0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a459a2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a459a2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35416/0 (socket says 172.21.15.134:35416) 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a459a2700 1 -- 172.21.15.134:0/3601509664 learned_addr learned my addr 172.21.15.134:0/3601509664 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a459a2700 1 -- 172.21.15.134:0/3601509664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c005040 con 0x7f6a400f51a0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a459a2700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f6a3c013510 tx=0x7f6a3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a3c009320 con 0x7f6a400f51a0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a40060220 con 0x7f6a400f51a0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a3c013f00 con 0x7f6a400f51a0 2023-11-16T01:14:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a3c014dc0 con 0x7f6a400f51a0 2023-11-16T01:14:36.083 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.079+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4010baf0 con 0x7f6a400f51a0 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.080+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a3c0095f0 con 0x7f6a400f51a0 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.080+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a40045ad0 con 0x7f6a400f51a0 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.080+0000 7f6a367fc700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 0x7f6a2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.080+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a3c041000 con 0x7f6a400f51a0 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.080+0000 7f6a451a1700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 0x7f6a2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:36.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.081+0000 7f6a451a1700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 0x7f6a2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f6a300096a0 tx=0x7f6a30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:36.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.086+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a3c01c790 con 0x7f6a400f51a0 2023-11-16T01:14:36.296 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.295+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a4010c2b0 con 0x7f6a400f51a0 2023-11-16T01:14:36.297 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.296+0000 7f6a367fc700 1 -- 172.21.15.134:0/3601509664 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a3c010230 con 0x7f6a400f51a0 2023-11-16T01:14:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:36.298 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:36.299 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 msgr2=0x7f6a2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 0x7f6a2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f6a300096a0 tx=0x7f6a30006b40).stop 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 msgr2=0x7f6a4010b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f6a3c013510 tx=0x7f6a3c00b040).stop 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 shutdown_connections 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f6a2c02ff60 0x7f6a2c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:36.300 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 --2- 172.21.15.134:0/3601509664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a400f51a0 0x7f6a4010b600 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:36.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 >> 172.21.15.134:0/3601509664 conn(0x7f6a400f09a0 msgr2=0x7f6a400f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:36.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 shutdown_connections 2023-11-16T01:14:36.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:36.297+0000 7f6a469a4700 1 -- 172.21.15.134:0/3601509664 wait complete. 2023-11-16T01:14:36.301 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:37.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:36.297103+0000 mon.smithi133 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.134:0/3601509664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:37.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:37.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:38.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:35.987362+0000 mgr.smithi133.xdyttl (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 -- 172.21.15.134:0/4021703918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 msgr2=0x7f75600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 --2- 172.21.15.134:0/4021703918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f75600f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f7554009fb0 tx=0x7f7554009b30).stop 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 -- 172.21.15.134:0/4021703918 shutdown_connections 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 --2- 172.21.15.134:0/4021703918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f75600f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 -- 172.21.15.134:0/4021703918 >> 172.21.15.134:0/4021703918 conn(0x7f75600f0b70 msgr2=0x7f75600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:38.912 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 -- 172.21.15.134:0/4021703918 shutdown_connections 2023-11-16T01:14:38.913 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.911+0000 7f7566d1d700 1 -- 172.21.15.134:0/4021703918 wait complete. 2023-11-16T01:14:38.913 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7566d1d700 1 Processor -- start 2023-11-16T01:14:38.913 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7566d1d700 1 -- start start 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7566d1d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7566d1d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560060220 con 0x7f75600f5410 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7564ab9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7564ab9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:35426/0 (socket says 172.21.15.134:35426) 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7564ab9700 1 -- 172.21.15.134:0/1382404675 learned_addr learned my addr 172.21.15.134:0/1382404675 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.912+0000 7f7564ab9700 1 -- 172.21.15.134:0/1382404675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554005040 con 0x7f75600f5410 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f7564ab9700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7554013510 tx=0x7f755400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:38.914 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7554009320 con 0x7f75600f5410 2023-11-16T01:14:38.915 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560060420 con 0x7f75600f5410 2023-11-16T01:14:38.915 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7554013f00 con 0x7f75600f5410 2023-11-16T01:14:38.915 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7554014d90 con 0x7f75600f5410 2023-11-16T01:14:38.915 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.913+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7560109ad0 con 0x7f75600f5410 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.914+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75540263a0 con 0x7f75600f5410 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.914+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560059100 con 0x7f75600f5410 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.914+0000 7f755d7fa700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 0x7f7550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.914+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75540412f0 con 0x7f75600f5410 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.914+0000 7f755ffff700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 0x7f7550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:38.916 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.915+0000 7f755ffff700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 0x7f7550032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f754c0096a0 tx=0x7f754c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:38.920 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:38.919+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7554012510 con 0x7f75600f5410 2023-11-16T01:14:39.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.124+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7560045ad0 con 0x7f75600f5410 2023-11-16T01:14:39.125 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.125+0000 7f755d7fa700 1 -- 172.21.15.134:0/1382404675 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7554012330 con 0x7f75600f5410 2023-11-16T01:14:39.126 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:39.126 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:39.127 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 msgr2=0x7f7550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 0x7f7550032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f754c0096a0 tx=0x7f754c006b40).stop 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 msgr2=0x7f7560109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7554013510 tx=0x7f755400b040).stop 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 shutdown_connections 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f755002ff60 0x7f7550032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 --2- 172.21.15.134:0/1382404675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75600f5410 0x7f7560109420 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.126+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 >> 172.21.15.134:0/1382404675 conn(0x7f75600f0b70 msgr2=0x7f75600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.127+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 shutdown_connections 2023-11-16T01:14:39.128 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:39.127+0000 7f7566d1d700 1 -- 172.21.15.134:0/1382404675 wait complete. 2023-11-16T01:14:39.129 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:37.987679+0000 mgr.smithi133.xdyttl (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:39.125418+0000 mon.smithi133 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.134:0/1382404675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:40.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:40.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:41.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 -- 172.21.15.134:0/1178225143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 msgr2=0x7f1e8c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:41.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/1178225143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c0f90f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f1e80009fb0 tx=0x7f1e80009b30).stop 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 -- 172.21.15.134:0/1178225143 shutdown_connections 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/1178225143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c0f90f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 -- 172.21.15.134:0/1178225143 >> 172.21.15.134:0/1178225143 conn(0x7f1e8c0f0b70 msgr2=0x7f1e8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 -- 172.21.15.134:0/1178225143 shutdown_connections 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 -- 172.21.15.134:0/1178225143 wait complete. 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.748+0000 7f1e90e8c700 1 Processor -- start 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e90e8c700 1 -- start start 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e90e8c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e90e8c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c106bd0 con 0x7f1e8c060280 2023-11-16T01:14:41.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e8a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e8a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49064/0 (socket says 172.21.15.134:49064) 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e8a59c700 1 -- 172.21.15.134:0/4193822520 learned_addr learned my addr 172.21.15.134:0/4193822520 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e8a59c700 1 -- 172.21.15.134:0/4193822520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80005040 con 0x7f1e8c060280 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e8a59c700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1e80013510 tx=0x7f1e8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e80009320 con 0x7f1e8c060280 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e80013f00 con 0x7f1e8c060280 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c106dd0 con 0x7f1e8c060280 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.749+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e80014dc0 con 0x7f1e8c060280 2023-11-16T01:14:41.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.750+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c107260 con 0x7f1e8c060280 2023-11-16T01:14:41.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.750+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e800095f0 con 0x7f1e8c060280 2023-11-16T01:14:41.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.750+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c045ad0 con 0x7f1e8c060280 2023-11-16T01:14:41.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.750+0000 7f1e7affd700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:41.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.750+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e800410c0 con 0x7f1e8c060280 2023-11-16T01:14:41.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.751+0000 7f1e89d9b700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:41.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.751+0000 7f1e89d9b700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 0x7f1e74032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:41.755 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.755+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e8001c790 con 0x7f1e8c060280 2023-11-16T01:14:41.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.960+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e8c108110 con 0x7f1e8c060280 2023-11-16T01:14:41.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.960+0000 7f1e7affd700 1 -- 172.21.15.134:0/4193822520 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e80010230 con 0x7f1e8c060280 2023-11-16T01:14:41.961 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:41.961 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:41.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 msgr2=0x7f1e74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 0x7f1e74032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).stop 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 msgr2=0x7f1e8c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1e80013510 tx=0x7f1e8000b040).stop 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 shutdown_connections 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1e7402ff60 0x7f1e74032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 --2- 172.21.15.134:0/4193822520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e8c060280 0x7f1e8c1066e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 >> 172.21.15.134:0/4193822520 conn(0x7f1e8c0f0b70 msgr2=0x7f1e8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:41.963 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 shutdown_connections 2023-11-16T01:14:41.964 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:41.962+0000 7f1e90e8c700 1 -- 172.21.15.134:0/4193822520 wait complete. 2023-11-16T01:14:41.964 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:42.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:39.987998+0000 mgr.smithi133.xdyttl (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:43.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:42 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:41.960876+0000 mon.smithi133 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.134:0/4193822520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:43.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:43.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:44.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:41.988376+0000 mgr.smithi133.xdyttl (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:44.592 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 -- 172.21.15.134:0/1582554427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 msgr2=0x7f5fc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/1582554427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc40f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f5fb4009fb0 tx=0x7f5fb4009b30).stop 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 -- 172.21.15.134:0/1582554427 shutdown_connections 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/1582554427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc40f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 -- 172.21.15.134:0/1582554427 >> 172.21.15.134:0/1582554427 conn(0x7f5fc40f0b70 msgr2=0x7f5fc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 -- 172.21.15.134:0/1582554427 shutdown_connections 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.591+0000 7f5fcbefc700 1 -- 172.21.15.134:0/1582554427 wait complete. 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fcbefc700 1 Processor -- start 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fcbefc700 1 -- start start 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fcbefc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:44.593 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fcbefc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc4060240 con 0x7f5fc40f5410 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fc9c98700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fc9c98700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49092/0 (socket says 172.21.15.134:49092) 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.592+0000 7f5fc9c98700 1 -- 172.21.15.134:0/2514444069 learned_addr learned my addr 172.21.15.134:0/2514444069 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fc9c98700 1 -- 172.21.15.134:0/2514444069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb4005040 con 0x7f5fc40f5410 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fc9c98700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f5fb4013510 tx=0x7f5fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb4009320 con 0x7f5fc40f5410 2023-11-16T01:14:44.595 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fb4013f00 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc4060440 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fb4014d90 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.593+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc4109ae0 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.594+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fb40263a0 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.594+0000 7f5fba7fc700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 0x7f5fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.594+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fb40410c0 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.594+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc4045ad0 con 0x7f5fc40f5410 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.594+0000 7f5fc9497700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 0x7f5fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:44.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.595+0000 7f5fc9497700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 0x7f5fb0032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5fc00096a0 tx=0x7f5fc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:44.600 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.599+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fb4012920 con 0x7f5fc40f5410 2023-11-16T01:14:44.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.805+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fc410a230 con 0x7f5fc40f5410 2023-11-16T01:14:44.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.806+0000 7f5fba7fc700 1 -- 172.21.15.134:0/2514444069 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5fb402b020 con 0x7f5fc40f5410 2023-11-16T01:14:44.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:44.806 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 msgr2=0x7f5fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 0x7f5fb0032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5fc00096a0 tx=0x7f5fc0006b40).stop 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 msgr2=0x7f5fc4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f5fb4013510 tx=0x7f5fb400b040).stop 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 shutdown_connections 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5fb002ff60 0x7f5fb0032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 --2- 172.21.15.134:0/2514444069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5fc40f5410 0x7f5fc4109430 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 >> 172.21.15.134:0/2514444069 conn(0x7f5fc40f0b70 msgr2=0x7f5fc40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 shutdown_connections 2023-11-16T01:14:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:44.807+0000 7f5fcbefc700 1 -- 172.21.15.134:0/2514444069 wait complete. 2023-11-16T01:14:44.809 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:46.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:43.988736+0000 mgr.smithi133.xdyttl (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:46.179 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:44.806480+0000 mon.smithi133 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.134:0/2514444069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:46.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:46.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:47.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 -- 172.21.15.134:0/4095131667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 msgr2=0x7ff9140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:47.411 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 --2- 172.21.15.134:0/4095131667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff9140f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff90c009960 tx=0x7ff90c004d10).stop 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 -- 172.21.15.134:0/4095131667 shutdown_connections 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 --2- 172.21.15.134:0/4095131667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff9140f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 -- 172.21.15.134:0/4095131667 >> 172.21.15.134:0/4095131667 conn(0x7ff9140f0b70 msgr2=0x7ff9140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 -- 172.21.15.134:0/4095131667 shutdown_connections 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.408+0000 7ff9195d7700 1 -- 172.21.15.134:0/4095131667 wait complete. 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff9195d7700 1 Processor -- start 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff9195d7700 1 -- start start 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff9195d7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff9195d7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff914106c20 con 0x7ff9140f5410 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff912ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff912ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49110/0 (socket says 172.21.15.134:49110) 2023-11-16T01:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff912ffd700 1 -- 172.21.15.134:0/3451767834 learned_addr learned my addr 172.21.15.134:0/3451767834 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff912ffd700 1 -- 172.21.15.134:0/3451767834 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c009640 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.409+0000 7ff912ffd700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff90c004b70 tx=0x7ff90c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c018070 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff914106e20 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff90c00b870 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff90c01c560 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9141072b0 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff90c013630 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.410+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff914045ad0 con 0x7ff9140f5410 2023-11-16T01:14:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.411+0000 7ff8fb7fe700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 0x7ff8fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:47.414 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.411+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff90c0413d0 con 0x7ff9140f5410 2023-11-16T01:14:47.414 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.411+0000 7ff9127fc700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 0x7ff8fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:47.414 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.411+0000 7ff9127fc700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 0x7ff8fc032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff9080096a0 tx=0x7ff908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:47.416 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.415+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff90c004810 con 0x7ff9140f5410 2023-11-16T01:14:47.622 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.622+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9140fbf90 con 0x7ff9140f5410 2023-11-16T01:14:47.623 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.622+0000 7ff8fb7fe700 1 -- 172.21.15.134:0/3451767834 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff90c012380 con 0x7ff9140f5410 2023-11-16T01:14:47.623 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:47.623 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 msgr2=0x7ff8fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 0x7ff8fc032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff9080096a0 tx=0x7ff908006b40).stop 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 msgr2=0x7ff914106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff90c004b70 tx=0x7ff90c004d10).stop 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 shutdown_connections 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff8fc02ff60 0x7ff8fc032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 --2- 172.21.15.134:0/3451767834 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9140f5410 0x7ff914106730 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 >> 172.21.15.134:0/3451767834 conn(0x7ff9140f0b70 msgr2=0x7ff9140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 shutdown_connections 2023-11-16T01:14:47.625 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:47.624+0000 7ff9195d7700 1 -- 172.21.15.134:0/3451767834 wait complete. 2023-11-16T01:14:47.626 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:45.989107+0000 mgr.smithi133.xdyttl (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:47.622877+0000 mon.smithi133 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.134:0/3451767834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:49.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:49.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:50.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:47.989458+0000 mgr.smithi133.xdyttl (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:50.239 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 -- 172.21.15.134:0/288442394 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 msgr2=0x7f766c0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:50.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 --2- 172.21.15.134:0/288442394 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c0f4fa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f7660009fb0 tx=0x7f7660009b30).stop 2023-11-16T01:14:50.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 -- 172.21.15.134:0/288442394 shutdown_connections 2023-11-16T01:14:50.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 --2- 172.21.15.134:0/288442394 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c0f4fa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:50.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 -- 172.21.15.134:0/288442394 >> 172.21.15.134:0/288442394 conn(0x7f766c0f02e0 msgr2=0x7f766c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:50.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 -- 172.21.15.134:0/288442394 shutdown_connections 2023-11-16T01:14:50.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.230+0000 7f7672998700 1 -- 172.21.15.134:0/288442394 wait complete. 2023-11-16T01:14:50.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f7672998700 1 Processor -- start 2023-11-16T01:14:50.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f7672998700 1 -- start start 2023-11-16T01:14:50.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f7672998700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:50.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f7672998700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c108350 con 0x7f766c0f4b80 2023-11-16T01:14:50.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f766bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:50.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f766bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49750/0 (socket says 172.21.15.134:49750) 2023-11-16T01:14:50.245 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.231+0000 7f766bfff700 1 -- 172.21.15.134:0/3166684135 learned_addr learned my addr 172.21.15.134:0/3166684135 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:50.245 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f766bfff700 1 -- 172.21.15.134:0/3166684135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660005040 con 0x7f766c0f4b80 2023-11-16T01:14:50.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f766bfff700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f7660013510 tx=0x7f766000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:50.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7660009320 con 0x7f766c0f4b80 2023-11-16T01:14:50.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7660013f00 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c108550 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7660014d90 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.232+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c108960 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.233+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76600095f0 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.233+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c059100 con 0x7f766c0f4b80 2023-11-16T01:14:50.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.233+0000 7f7668ff9700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 0x7f7654032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:50.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.233+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7660040fc0 con 0x7f766c0f4b80 2023-11-16T01:14:50.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.233+0000 7f766b7fe700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 0x7f7654032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:50.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.234+0000 7f766b7fe700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 0x7f7654032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f765c0096a0 tx=0x7f765c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:50.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.238+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7660014420 con 0x7f766c0f4b80 2023-11-16T01:14:50.442 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.442+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f766c0fa5d0 con 0x7f766c0f4b80 2023-11-16T01:14:50.443 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.442+0000 7f7668ff9700 1 -- 172.21.15.134:0/3166684135 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7660010230 con 0x7f766c0f4b80 2023-11-16T01:14:50.444 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:50.444 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 msgr2=0x7f7654032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 0x7f7654032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f765c0096a0 tx=0x7f765c006b40).stop 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 msgr2=0x7f766c107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f7660013510 tx=0x7f766000b040).stop 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 shutdown_connections 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f765402ff60 0x7f7654032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 --2- 172.21.15.134:0/3166684135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f766c0f4b80 0x7f766c107e60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 >> 172.21.15.134:0/3166684135 conn(0x7f766c0f02e0 msgr2=0x7f766c0f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 shutdown_connections 2023-11-16T01:14:50.445 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:50.444+0000 7f7672998700 1 -- 172.21.15.134:0/3166684135 wait complete. 2023-11-16T01:14:50.446 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:51.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:50 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:50.442886+0000 mon.smithi133 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.134:0/3166684135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:51.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:51.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:52.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:49.989779+0000 mgr.smithi133.xdyttl (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:53.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.019+0000 7f698cca9700 1 -- 172.21.15.134:0/425951343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 msgr2=0x7f69880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.019+0000 7f698cca9700 1 --2- 172.21.15.134:0/425951343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69880f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f6978009fb0 tx=0x7f6978009b30).stop 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 -- 172.21.15.134:0/425951343 shutdown_connections 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 --2- 172.21.15.134:0/425951343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69880f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 -- 172.21.15.134:0/425951343 >> 172.21.15.134:0/425951343 conn(0x7f69880f09a0 msgr2=0x7f69880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 -- 172.21.15.134:0/425951343 shutdown_connections 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 -- 172.21.15.134:0/425951343 wait complete. 2023-11-16T01:14:53.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 Processor -- start 2023-11-16T01:14:53.022 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.020+0000 7f698cca9700 1 -- start start 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f698cca9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f698cca9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6988060240 con 0x7f69880f51a0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f69877fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f69877fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49768/0 (socket says 172.21.15.134:49768) 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f69877fe700 1 -- 172.21.15.134:0/404688145 learned_addr learned my addr 172.21.15.134:0/404688145 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f69877fe700 1 -- 172.21.15.134:0/404688145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978005040 con 0x7f69880f51a0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.021+0000 7f69877fe700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6978013510 tx=0x7f697800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6978009320 con 0x7f69880f51a0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6978013f00 con 0x7f69880f51a0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6988060440 con 0x7f69880f51a0 2023-11-16T01:14:53.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6978014dc0 con 0x7f69880f51a0 2023-11-16T01:14:53.024 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6988109860 con 0x7f69880f51a0 2023-11-16T01:14:53.024 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69780095f0 con 0x7f69880f51a0 2023-11-16T01:14:53.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.022+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6988045ad0 con 0x7f69880f51a0 2023-11-16T01:14:53.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.023+0000 7f696ffff700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 0x7f6970032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:53.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.023+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69780410c0 con 0x7f69880f51a0 2023-11-16T01:14:53.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.023+0000 7f6986ffd700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 0x7f6970032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:53.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.023+0000 7f6986ffd700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 0x7f6970032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f697c0096a0 tx=0x7f697c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:53.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.028+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6978014420 con 0x7f69880f51a0 2023-11-16T01:14:53.233 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.233+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f698810a080 con 0x7f69880f51a0 2023-11-16T01:14:53.234 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.234+0000 7f696ffff700 1 -- 172.21.15.134:0/404688145 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6978010230 con 0x7f69880f51a0 2023-11-16T01:14:53.235 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:53.235 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 msgr2=0x7f6970032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 0x7f6970032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f697c0096a0 tx=0x7f697c006b40).stop 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 msgr2=0x7f69881091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6978013510 tx=0x7f697800b040).stop 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 shutdown_connections 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f697002ff60 0x7f6970032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 --2- 172.21.15.134:0/404688145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69880f51a0 0x7f69881091b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:53.236 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 >> 172.21.15.134:0/404688145 conn(0x7f69880f09a0 msgr2=0x7f69880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:53.237 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 shutdown_connections 2023-11-16T01:14:53.237 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:53.235+0000 7f698cca9700 1 -- 172.21.15.134:0/404688145 wait complete. 2023-11-16T01:14:53.237 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:51.990136+0000 mgr.smithi133.xdyttl (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:54.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:53.234167+0000 mon.smithi133 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.134:0/404688145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:54.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:54.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:55.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 -- 172.21.15.134:0/2558160518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 msgr2=0x7fd0300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:55.775 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 --2- 172.21.15.134:0/2558160518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd0300f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fd020009fb0 tx=0x7fd020009b30).stop 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 -- 172.21.15.134:0/2558160518 shutdown_connections 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 --2- 172.21.15.134:0/2558160518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd0300f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 -- 172.21.15.134:0/2558160518 >> 172.21.15.134:0/2558160518 conn(0x7fd0300f0b70 msgr2=0x7fd0300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 -- 172.21.15.134:0/2558160518 shutdown_connections 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.772+0000 7fd037343700 1 -- 172.21.15.134:0/2558160518 wait complete. 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd037343700 1 Processor -- start 2023-11-16T01:14:55.776 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd037343700 1 -- start start 2023-11-16T01:14:55.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd037343700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:55.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd037343700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd030105b20 con 0x7fd0300f5410 2023-11-16T01:14:55.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd0350df700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:55.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd0350df700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49774/0 (socket says 172.21.15.134:49774) 2023-11-16T01:14:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.773+0000 7fd0350df700 1 -- 172.21.15.134:0/367872934 learned_addr learned my addr 172.21.15.134:0/367872934 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd0350df700 1 -- 172.21.15.134:0/367872934 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd020005040 con 0x7fd0300f5410 2023-11-16T01:14:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd0350df700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fd020013510 tx=0x7fd02000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd020009320 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd030105d20 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd020013f00 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd020014d90 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.774+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0301061b0 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.775+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd020014420 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.775+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd030045ad0 con 0x7fd0300f5410 2023-11-16T01:14:55.819 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.775+0000 7fd025ffb700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 0x7fd01c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:55.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.775+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd020041080 con 0x7fd0300f5410 2023-11-16T01:14:55.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.775+0000 7fd0348de700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 0x7fd01c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:55.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.776+0000 7fd0348de700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 0x7fd01c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd02c0096a0 tx=0x7fd02c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:55.820 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.780+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0200095f0 con 0x7fd0300f5410 2023-11-16T01:14:55.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.986+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0300fae60 con 0x7fd0300f5410 2023-11-16T01:14:55.988 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.987+0000 7fd025ffb700 1 -- 172.21.15.134:0/367872934 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd02001db70 con 0x7fd0300f5410 2023-11-16T01:14:55.989 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 msgr2=0x7fd01c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 0x7fd01c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd02c0096a0 tx=0x7fd02c006b40).stop 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 msgr2=0x7fd030105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fd020013510 tx=0x7fd02000b040).stop 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 shutdown_connections 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd01c02ff60 0x7fd01c032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 --2- 172.21.15.134:0/367872934 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0300f5410 0x7fd030105630 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 >> 172.21.15.134:0/367872934 conn(0x7fd0300f0b70 msgr2=0x7fd0300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:55.991 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 shutdown_connections 2023-11-16T01:14:55.991 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:55.988+0000 7fd037343700 1 -- 172.21.15.134:0/367872934 wait complete. 2023-11-16T01:14:55.991 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:14:56.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:53.990513+0000 mgr.smithi133.xdyttl (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:57.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:55.988227+0000 mon.smithi133 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.134:0/367872934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:14:57.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:14:57.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:14:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:55.990797+0000 mgr.smithi133.xdyttl (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3431288124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 msgr2=0x7fc8a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3431288124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a40f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fc894009fb0 tx=0x7fc894009b30).stop 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3431288124 shutdown_connections 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3431288124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a40f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3431288124 >> 172.21.15.134:0/3431288124 conn(0x7fc8a40f0b70 msgr2=0x7fc8a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:58.551 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.547+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3431288124 shutdown_connections 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.548+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3431288124 wait complete. 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.548+0000 7fc8ac7df700 1 Processor -- start 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.548+0000 7fc8ac7df700 1 -- start start 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.548+0000 7fc8ac7df700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.548+0000 7fc8ac7df700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a4060240 con 0x7fc8a40f5410 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8aa57b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8aa57b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49796/0 (socket says 172.21.15.134:49796) 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8aa57b700 1 -- 172.21.15.134:0/3128002854 learned_addr learned my addr 172.21.15.134:0/3128002854 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8aa57b700 1 -- 172.21.15.134:0/3128002854 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc894005040 con 0x7fc8a40f5410 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8aa57b700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fc894013510 tx=0x7fc89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:58.552 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc894009320 con 0x7fc8a40f5410 2023-11-16T01:14:58.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a4060440 con 0x7fc8a40f5410 2023-11-16T01:14:58.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc894013f00 con 0x7fc8a40f5410 2023-11-16T01:14:58.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc894014d90 con 0x7fc8a40f5410 2023-11-16T01:14:58.553 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.549+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a4109ae0 con 0x7fc8a40f5410 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.550+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8940095f0 con 0x7fc8a40f5410 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.550+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a4045ad0 con 0x7fc8a40f5410 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.550+0000 7fc89affd700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 0x7fc890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.550+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc894041000 con 0x7fc8a40f5410 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.551+0000 7fc8a9d7a700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 0x7fc890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.551+0000 7fc8a9d7a700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 0x7fc890032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:14:58.557 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.555+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc894014420 con 0x7fc8a40f5410 2023-11-16T01:14:58.761 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.760+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8a40fae60 con 0x7fc8a40f5410 2023-11-16T01:14:58.762 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.760+0000 7fc89affd700 1 -- 172.21.15.134:0/3128002854 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc89401d7f0 con 0x7fc8a40f5410 2023-11-16T01:14:58.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:14:58.762 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:14:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 msgr2=0x7fc890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 0x7fc890032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).stop 2023-11-16T01:14:58.763 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 msgr2=0x7fc8a4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fc894013510 tx=0x7fc89400b040).stop 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 shutdown_connections 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc89002ff60 0x7fc890032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 --2- 172.21.15.134:0/3128002854 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8a40f5410 0x7fc8a4109430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 >> 172.21.15.134:0/3128002854 conn(0x7fc8a40f0b70 msgr2=0x7fc8a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 shutdown_connections 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:14:58.762+0000 7fc8ac7df700 1 -- 172.21.15.134:0/3128002854 wait complete. 2023-11-16T01:14:58.764 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:57.991122+0000 mgr.smithi133.xdyttl (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:14:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:14:58.761790+0000 mon.smithi133 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.134:0/3128002854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:00.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:00.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:01.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 -- 172.21.15.134:0/1731170491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 msgr2=0x7f425c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:01.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 --2- 172.21.15.134:0/1731170491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f4250009fb0 tx=0x7f4250009b30).stop 2023-11-16T01:15:01.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 -- 172.21.15.134:0/1731170491 shutdown_connections 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 --2- 172.21.15.134:0/1731170491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 -- 172.21.15.134:0/1731170491 >> 172.21.15.134:0/1731170491 conn(0x7f425c0f0b70 msgr2=0x7f425c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 -- 172.21.15.134:0/1731170491 shutdown_connections 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 -- 172.21.15.134:0/1731170491 wait complete. 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.305+0000 7f4262c86700 1 Processor -- start 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4262c86700 1 -- start start 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4262c86700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4262c86700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c107210 con 0x7f425c0f5410 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4260a22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4260a22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58606/0 (socket says 172.21.15.134:58606) 2023-11-16T01:15:01.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4260a22700 1 -- 172.21.15.134:0/551579629 learned_addr learned my addr 172.21.15.134:0/551579629 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4260a22700 1 -- 172.21.15.134:0/551579629 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250005040 con 0x7f425c0f5410 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.306+0000 7f4260a22700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f4250013510 tx=0x7f425000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4250009320 con 0x7f425c0f5410 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c060220 con 0x7f425c0f5410 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4250013f00 con 0x7f425c0f5410 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4250014d90 con 0x7f425c0f5410 2023-11-16T01:15:01.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c10bd10 con 0x7f425c0f5410 2023-11-16T01:15:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4250014420 con 0x7f425c0f5410 2023-11-16T01:15:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.307+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c045ad0 con 0x7f425c0f5410 2023-11-16T01:15:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.308+0000 7f42597fa700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.308+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4250041000 con 0x7f425c0f5410 2023-11-16T01:15:01.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.308+0000 7f425bfff700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:01.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.308+0000 7f425bfff700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f42480096a0 tx=0x7f4248006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:01.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.312+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42500095f0 con 0x7f425c0f5410 2023-11-16T01:15:01.519 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.518+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f425c0fae60 con 0x7f425c0f5410 2023-11-16T01:15:01.520 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.518+0000 7f42597fa700 1 -- 172.21.15.134:0/551579629 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f425001db70 con 0x7f425c0f5410 2023-11-16T01:15:01.520 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:01.520 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 msgr2=0x7f424c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f42480096a0 tx=0x7f4248006b40).stop 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 msgr2=0x7f425c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f4250013510 tx=0x7f425000b040).stop 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 shutdown_connections 2023-11-16T01:15:01.521 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:01.522 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 --2- 172.21.15.134:0/551579629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f425c0f5410 0x7f425c10b820 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:01.522 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 >> 172.21.15.134:0/551579629 conn(0x7f425c0f0b70 msgr2=0x7f425c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:01.522 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 shutdown_connections 2023-11-16T01:15:01.522 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:01.520+0000 7f4262c86700 1 -- 172.21.15.134:0/551579629 wait complete. 2023-11-16T01:15:01.522 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:14:59.991473+0000 mgr.smithi133.xdyttl (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:01.519751+0000 mon.smithi133 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.134:0/551579629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:02.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:02.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:01.991793+0000 mgr.smithi133.xdyttl (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:04.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 -- 172.21.15.134:0/451264160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 msgr2=0x7f7fe40f90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 --2- 172.21.15.134:0/451264160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe40f90b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f7fd4009fb0 tx=0x7f7fd4009b30).stop 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 -- 172.21.15.134:0/451264160 shutdown_connections 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 --2- 172.21.15.134:0/451264160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe40f90b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 -- 172.21.15.134:0/451264160 >> 172.21.15.134:0/451264160 conn(0x7f7fe40f0a90 msgr2=0x7f7fe40f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 -- 172.21.15.134:0/451264160 shutdown_connections 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.146+0000 7f7fea50c700 1 -- 172.21.15.134:0/451264160 wait complete. 2023-11-16T01:15:04.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.147+0000 7f7fea50c700 1 Processor -- start 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.147+0000 7f7fea50c700 1 -- start start 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fea50c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fea50c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4106e30 con 0x7f7fe40601e0 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58634/0 (socket says 172.21.15.134:58634) 2023-11-16T01:15:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe3fff700 1 -- 172.21.15.134:0/550326422 learned_addr learned my addr 172.21.15.134:0/550326422 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe3fff700 1 -- 172.21.15.134:0/550326422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4005040 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe3fff700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7fd4004dd0 tx=0x7f7fd4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fd4018070 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe4107030 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fd4013410 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fd401c3c0 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.148+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe41074c0 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.149+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fd401d420 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.149+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe4045ad0 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.149+0000 7f7fe0ff9700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 0x7f7fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.149+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fd4041390 con 0x7f7fe40601e0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.149+0000 7f7fe37fe700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 0x7f7fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:04.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.150+0000 7f7fe37fe700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 0x7f7fcc032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7fd80096a0 tx=0x7f7fd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:04.156 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.154+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fd401d820 con 0x7f7fe40601e0 2023-11-16T01:15:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.360+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fe405ea50 con 0x7f7fe40601e0 2023-11-16T01:15:04.362 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.361+0000 7f7fe0ff9700 1 -- 172.21.15.134:0/550326422 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7fd401b370 con 0x7f7fe40601e0 2023-11-16T01:15:04.362 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:04.362 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 msgr2=0x7f7fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 0x7f7fcc032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7fd80096a0 tx=0x7f7fd8006b40).stop 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 msgr2=0x7f7fe4106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7fd4004dd0 tx=0x7f7fd4009b30).stop 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 shutdown_connections 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7fcc02ff60 0x7f7fcc032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 --2- 172.21.15.134:0/550326422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7fe40601e0 0x7f7fe4106940 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 >> 172.21.15.134:0/550326422 conn(0x7f7fe40f0a90 msgr2=0x7f7fe40f1570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 shutdown_connections 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:04.362+0000 7f7fea50c700 1 -- 172.21.15.134:0/550326422 wait complete. 2023-11-16T01:15:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:05.033 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:04 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:04.362195+0000 mon.smithi133 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.134:0/550326422' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:05.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:05.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:03.992145+0000 mgr.smithi133.xdyttl (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 -- 172.21.15.134:0/3963186632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 msgr2=0x7fe9f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 --2- 172.21.15.134:0/3963186632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f80f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fe9f4009fb0 tx=0x7fe9f4009b30).stop 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 -- 172.21.15.134:0/3963186632 shutdown_connections 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 --2- 172.21.15.134:0/3963186632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f80f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 -- 172.21.15.134:0/3963186632 >> 172.21.15.134:0/3963186632 conn(0x7fe9f80f1c70 msgr2=0x7fe9f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:07.036 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 -- 172.21.15.134:0/3963186632 shutdown_connections 2023-11-16T01:15:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.033+0000 7fe9ff651700 1 -- 172.21.15.134:0/3963186632 wait complete. 2023-11-16T01:15:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9ff651700 1 Processor -- start 2023-11-16T01:15:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9ff651700 1 -- start start 2023-11-16T01:15:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9ff651700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9ff651700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8106c20 con 0x7fe9f80f6510 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9fd3ed700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9fd3ed700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:58654/0 (socket says 172.21.15.134:58654) 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9fd3ed700 1 -- 172.21.15.134:0/1556050809 learned_addr learned my addr 172.21.15.134:0/1556050809 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9fd3ed700 1 -- 172.21.15.134:0/1556050809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f4005040 con 0x7fe9f80f6510 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.034+0000 7fe9fd3ed700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe9f4013510 tx=0x7fe9f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.035+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f4009320 con 0x7fe9f80f6510 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.035+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f8106e20 con 0x7fe9f80f6510 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.035+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9f4013f00 con 0x7fe9f80f6510 2023-11-16T01:15:07.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.035+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f4014d90 con 0x7fe9f80f6510 2023-11-16T01:15:07.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.035+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f81072b0 con 0x7fe9f80f6510 2023-11-16T01:15:07.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.036+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9f40095f0 con 0x7fe9f80f6510 2023-11-16T01:15:07.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.036+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f8045ad0 con 0x7fe9f80f6510 2023-11-16T01:15:07.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.036+0000 7fe9edffb700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:07.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.036+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9f4040fc0 con 0x7fe9f80f6510 2023-11-16T01:15:07.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.036+0000 7fe9fcbec700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:07.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.037+0000 7fe9fcbec700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe9e80096a0 tx=0x7fe9e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:07.042 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.041+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9f4014420 con 0x7fe9f80f6510 2023-11-16T01:15:07.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.246+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9f80fbf60 con 0x7fe9f80f6510 2023-11-16T01:15:07.248 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.247+0000 7fe9edffb700 1 -- 172.21.15.134:0/1556050809 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9f4010230 con 0x7fe9f80f6510 2023-11-16T01:15:07.249 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:07.249 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:07.250 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 msgr2=0x7fe9e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe9e80096a0 tx=0x7fe9e8006b40).stop 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 msgr2=0x7fe9f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe9f4013510 tx=0x7fe9f400b040).stop 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 shutdown_connections 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 --2- 172.21.15.134:0/1556050809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9f80f6510 0x7fe9f8106730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 >> 172.21.15.134:0/1556050809 conn(0x7fe9f80f1c70 msgr2=0x7fe9f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 shutdown_connections 2023-11-16T01:15:07.251 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:07.249+0000 7fe9ff651700 1 -- 172.21.15.134:0/1556050809 wait complete. 2023-11-16T01:15:07.252 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:05.992425+0000 mgr.smithi133.xdyttl (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:07.248634+0000 mon.smithi133 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.134:0/1556050809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:08.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:08.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:09.874 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.869+0000 7f85f87b9700 1 -- 172.21.15.134:0/894399200 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 msgr2=0x7f85f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:09.874 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.869+0000 7f85f87b9700 1 --2- 172.21.15.134:0/894399200 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f00f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f85e0009fb0 tx=0x7f85e0009b30).stop 2023-11-16T01:15:09.874 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- 172.21.15.134:0/894399200 shutdown_connections 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 --2- 172.21.15.134:0/894399200 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f00f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- 172.21.15.134:0/894399200 >> 172.21.15.134:0/894399200 conn(0x7f85f00f1c70 msgr2=0x7f85f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- 172.21.15.134:0/894399200 shutdown_connections 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- 172.21.15.134:0/894399200 wait complete. 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 Processor -- start 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- start start 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f87b9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0106c20 con 0x7f85f00f6510 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f6555700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:09.875 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f6555700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55558/0 (socket says 172.21.15.134:55558) 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.870+0000 7f85f6555700 1 -- 172.21.15.134:0/2476796038 learned_addr learned my addr 172.21.15.134:0/2476796038 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85f6555700 1 -- 172.21.15.134:0/2476796038 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e0005040 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85f6555700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f85e0013510 tx=0x7f85e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e0009320 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f0106e20 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85e0013f00 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e0014d90 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.871+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f01072b0 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.872+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85e00095f0 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.872+0000 7f85e6ffd700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 0x7f85dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.872+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85e0041000 con 0x7f85f00f6510 2023-11-16T01:15:09.876 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.872+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f0045ad0 con 0x7f85f00f6510 2023-11-16T01:15:09.877 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.872+0000 7f85f5d54700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 0x7f85dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:09.877 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.873+0000 7f85f5d54700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 0x7f85dc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:09.878 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:09.877+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85e0014420 con 0x7f85f00f6510 2023-11-16T01:15:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:07.992740+0000 mgr.smithi133.xdyttl (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:10.085 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.084+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85f00fc050 con 0x7f85f00f6510 2023-11-16T01:15:10.086 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.084+0000 7f85e6ffd700 1 -- 172.21.15.134:0/2476796038 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85e002b020 con 0x7f85f00f6510 2023-11-16T01:15:10.086 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:10.087 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:10.088 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 msgr2=0x7f85dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:10.088 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 0x7f85dc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).stop 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 msgr2=0x7f85f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f85e0013510 tx=0x7f85e000b040).stop 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 shutdown_connections 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85dc02ff60 0x7f85dc032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 --2- 172.21.15.134:0/2476796038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85f00f6510 0x7f85f0106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 >> 172.21.15.134:0/2476796038 conn(0x7f85f00f1c70 msgr2=0x7f85f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:10.089 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 shutdown_connections 2023-11-16T01:15:10.090 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:10.086+0000 7f85f87b9700 1 -- 172.21.15.134:0/2476796038 wait complete. 2023-11-16T01:15:10.090 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:11.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:10 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:10.085926+0000 mon.smithi133 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.134:0/2476796038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:11.505 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:11.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:09.993008+0000 mgr.smithi133.xdyttl (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:12.788 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 -- 172.21.15.134:0/369367739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 msgr2=0x7f0d780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:12.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/369367739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d780f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0d6c009fb0 tx=0x7f0d6c009b30).stop 2023-11-16T01:15:12.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 -- 172.21.15.134:0/369367739 shutdown_connections 2023-11-16T01:15:12.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/369367739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d780f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:12.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 -- 172.21.15.134:0/369367739 >> 172.21.15.134:0/369367739 conn(0x7f0d780f1c70 msgr2=0x7f0d780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:12.789 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 -- 172.21.15.134:0/369367739 shutdown_connections 2023-11-16T01:15:12.790 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 -- 172.21.15.134:0/369367739 wait complete. 2023-11-16T01:15:12.790 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.783+0000 7f0d7cb50700 1 Processor -- start 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7cb50700 1 -- start start 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7cb50700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7cb50700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78106c20 con 0x7f0d780f6510 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55588/0 (socket says 172.21.15.134:55588) 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7659c700 1 -- 172.21.15.134:0/1412012018 learned_addr learned my addr 172.21.15.134:0/1412012018 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7659c700 1 -- 172.21.15.134:0/1412012018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c005040 con 0x7f0d780f6510 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7659c700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0d6c013510 tx=0x7f0d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d6c009320 con 0x7f0d780f6510 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.784+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d78106e20 con 0x7f0d780f6510 2023-11-16T01:15:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d6c013f00 con 0x7f0d780f6510 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d6c014d90 con 0x7f0d780f6510 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d781072b0 con 0x7f0d780f6510 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d6c014420 con 0x7f0d780f6510 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d78045ad0 con 0x7f0d780f6510 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.785+0000 7f0d66ffd700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:12.797 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.786+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d6c041110 con 0x7f0d780f6510 2023-11-16T01:15:12.798 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.786+0000 7f0d75d9b700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:12.798 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.786+0000 7f0d75d9b700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 0x7f0d60032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:12.799 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.790+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d6c0095f0 con 0x7f0d780f6510 2023-11-16T01:15:13.001 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:12.999+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d780fbf60 con 0x7f0d780f6510 2023-11-16T01:15:13.002 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.000+0000 7f0d66ffd700 1 -- 172.21.15.134:0/1412012018 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d6c01db70 con 0x7f0d780f6510 2023-11-16T01:15:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:13.003 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 msgr2=0x7f0d60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 0x7f0d60032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).stop 2023-11-16T01:15:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 msgr2=0x7f0d78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:13.004 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0d6c013510 tx=0x7f0d6c00b040).stop 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 shutdown_connections 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 --2- 172.21.15.134:0/1412012018 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d780f6510 0x7f0d78106730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 >> 172.21.15.134:0/1412012018 conn(0x7f0d780f1c70 msgr2=0x7f0d780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 shutdown_connections 2023-11-16T01:15:13.005 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:13.001+0000 7f0d7cb50700 1 -- 172.21.15.134:0/1412012018 wait complete. 2023-11-16T01:15:13.009 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:14.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:11.993260+0000 mgr.smithi133.xdyttl (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:14.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:13.001439+0000 mon.smithi133 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.134:0/1412012018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:14.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:14.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:15.536 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 -- 172.21.15.134:0/2082976690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 msgr2=0x7fe4700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 --2- 172.21.15.134:0/2082976690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe4700f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fe460009fb0 tx=0x7fe460009b30).stop 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 -- 172.21.15.134:0/2082976690 shutdown_connections 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 --2- 172.21.15.134:0/2082976690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe4700f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 -- 172.21.15.134:0/2082976690 >> 172.21.15.134:0/2082976690 conn(0x7fe4700f0b70 msgr2=0x7fe4700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 -- 172.21.15.134:0/2082976690 shutdown_connections 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.527+0000 7fe4786c4700 1 -- 172.21.15.134:0/2082976690 wait complete. 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe4786c4700 1 Processor -- start 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe4786c4700 1 -- start start 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe4786c4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe4786c4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470107240 con 0x7fe4700f5410 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe476460700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:15.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe476460700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55594/0 (socket says 172.21.15.134:55594) 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe476460700 1 -- 172.21.15.134:0/941804546 learned_addr learned my addr 172.21.15.134:0/941804546 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.528+0000 7fe476460700 1 -- 172.21.15.134:0/941804546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460005040 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe476460700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe460013510 tx=0x7fe46000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe460009320 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe470060220 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe460013f00 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe460014d90 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.529+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47010bd40 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4600095f0 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe470045ad0 con 0x7fe4700f5410 2023-11-16T01:15:15.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe466ffd700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 0x7fe45c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:15.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe460041000 con 0x7fe4700f5410 2023-11-16T01:15:15.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe475c5f700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 0x7fe45c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:15.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.530+0000 7fe475c5f700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 0x7fe45c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe46c0096a0 tx=0x7fe46c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:15.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.534+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe460014420 con 0x7fe4700f5410 2023-11-16T01:15:15.744 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.742+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4700fae60 con 0x7fe4700f5410 2023-11-16T01:15:15.744 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.742+0000 7fe466ffd700 1 -- 172.21.15.134:0/941804546 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe46001d7f0 con 0x7fe4700f5410 2023-11-16T01:15:15.745 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:15.746 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 msgr2=0x7fe45c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 0x7fe45c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe46c0096a0 tx=0x7fe46c006b40).stop 2023-11-16T01:15:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 msgr2=0x7fe47010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:15.747 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe460013510 tx=0x7fe46000b040).stop 2023-11-16T01:15:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 shutdown_connections 2023-11-16T01:15:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe45c02ff60 0x7fe45c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 --2- 172.21.15.134:0/941804546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe4700f5410 0x7fe47010b850 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 >> 172.21.15.134:0/941804546 conn(0x7fe4700f0b70 msgr2=0x7fe4700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:15.748 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 shutdown_connections 2023-11-16T01:15:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:15.744+0000 7fe4786c4700 1 -- 172.21.15.134:0/941804546 wait complete. 2023-11-16T01:15:15.749 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:13.993572+0000 mgr.smithi133.xdyttl (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:15.744043+0000 mon.smithi133 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.134:0/941804546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:17.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:17.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:18.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:15.993911+0000 mgr.smithi133.xdyttl (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:18.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/3674442959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 msgr2=0x7fc0a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:18.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/3674442959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a00f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fc094009fb0 tx=0x7fc094009b30).stop 2023-11-16T01:15:18.301 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/3674442959 shutdown_connections 2023-11-16T01:15:18.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/3674442959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a00f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:18.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/3674442959 >> 172.21.15.134:0/3674442959 conn(0x7fc0a00f0b70 msgr2=0x7fc0a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:18.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/3674442959 shutdown_connections 2023-11-16T01:15:18.302 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.299+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/3674442959 wait complete. 2023-11-16T01:15:18.305 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc0a4d5f700 1 Processor -- start 2023-11-16T01:15:18.305 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc0a4d5f700 1 -- start start 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc0a4d5f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc0a4d5f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0060280 con 0x7fc0a00f5410 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc09e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc09e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:55614/0 (socket says 172.21.15.134:55614) 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.300+0000 7fc09e59c700 1 -- 172.21.15.134:0/1788383833 learned_addr learned my addr 172.21.15.134:0/1788383833 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc09e59c700 1 -- 172.21.15.134:0/1788383833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094005040 con 0x7fc0a00f5410 2023-11-16T01:15:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc09e59c700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc094013510 tx=0x7fc09400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc094009320 con 0x7fc0a00f5410 2023-11-16T01:15:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc094013f00 con 0x7fc0a00f5410 2023-11-16T01:15:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a0060480 con 0x7fc0a00f5410 2023-11-16T01:15:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc094014d90 con 0x7fc0a00f5410 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.301+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a0107910 con 0x7fc0a00f5410 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.302+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0940095f0 con 0x7fc0a00f5410 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.302+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a0045ad0 con 0x7fc0a00f5410 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.302+0000 7fc08effd700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 0x7fc088032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.302+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc094041000 con 0x7fc0a00f5410 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.302+0000 7fc09dd9b700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 0x7fc088032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:18.309 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.303+0000 7fc09dd9b700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 0x7fc088032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:18.310 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.307+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc094014420 con 0x7fc0a00f5410 2023-11-16T01:15:18.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.510+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0a00fae60 con 0x7fc0a00f5410 2023-11-16T01:15:18.512 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.511+0000 7fc08effd700 1 -- 172.21.15.134:0/1788383833 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc09401d7f0 con 0x7fc0a00f5410 2023-11-16T01:15:18.513 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:18.513 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:18.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 msgr2=0x7fc088032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:18.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 0x7fc088032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).stop 2023-11-16T01:15:18.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 msgr2=0x7fc0a01071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:18.514 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc094013510 tx=0x7fc09400b040).stop 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 shutdown_connections 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc08802ff60 0x7fc088032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 --2- 172.21.15.134:0/1788383833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc0a00f5410 0x7fc0a01071e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 >> 172.21.15.134:0/1788383833 conn(0x7fc0a00f0b70 msgr2=0x7fc0a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 shutdown_connections 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:18.513+0000 7fc0a4d5f700 1 -- 172.21.15.134:0/1788383833 wait complete. 2023-11-16T01:15:18.515 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:19.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:18 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:18.512727+0000 mon.smithi133 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.134:0/1788383833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:19.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:19.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:20.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:17.994329+0000 mgr.smithi133.xdyttl (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 -- 172.21.15.134:0/840728078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 msgr2=0x7fec740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 --2- 172.21.15.134:0/840728078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec740f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fec68009fb0 tx=0x7fec68009b30).stop 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 -- 172.21.15.134:0/840728078 shutdown_connections 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 --2- 172.21.15.134:0/840728078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec740f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 -- 172.21.15.134:0/840728078 >> 172.21.15.134:0/840728078 conn(0x7fec740f1c70 msgr2=0x7fec740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 -- 172.21.15.134:0/840728078 shutdown_connections 2023-11-16T01:15:21.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.150+0000 7fec7ad63700 1 -- 172.21.15.134:0/840728078 wait complete. 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec7ad63700 1 Processor -- start 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec7ad63700 1 -- start start 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec7ad63700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec7ad63700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74106c20 con 0x7fec740f6510 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec78aff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec78aff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33294/0 (socket says 172.21.15.134:33294) 2023-11-16T01:15:21.153 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.151+0000 7fec78aff700 1 -- 172.21.15.134:0/3601601778 learned_addr learned my addr 172.21.15.134:0/3601601778 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:21.154 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec78aff700 1 -- 172.21.15.134:0/3601601778 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec68005040 con 0x7fec740f6510 2023-11-16T01:15:21.154 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec78aff700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fec68013510 tx=0x7fec6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec68009320 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec68013f00 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74106e20 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec68014d90 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.152+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec741072b0 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.153+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec680095f0 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.153+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74045ad0 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.153+0000 7fec717fa700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 0x7fec64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.153+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec680410c0 con 0x7fec740f6510 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.153+0000 7fec73fff700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 0x7fec64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.154+0000 7fec73fff700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 0x7fec64032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fec600096a0 tx=0x7fec60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:21.158 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.158+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec680263a0 con 0x7fec740f6510 2023-11-16T01:15:21.362 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.361+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec740fbf60 con 0x7fec740f6510 2023-11-16T01:15:21.363 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.362+0000 7fec717fa700 1 -- 172.21.15.134:0/3601601778 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec68010230 con 0x7fec740f6510 2023-11-16T01:15:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:21.363 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 msgr2=0x7fec64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:21.364 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 0x7fec64032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fec600096a0 tx=0x7fec60006b40).stop 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 msgr2=0x7fec74106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fec68013510 tx=0x7fec6800b040).stop 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 shutdown_connections 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fec6402ff60 0x7fec64032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 --2- 172.21.15.134:0/3601601778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fec740f6510 0x7fec74106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 >> 172.21.15.134:0/3601601778 conn(0x7fec740f1c70 msgr2=0x7fec740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 shutdown_connections 2023-11-16T01:15:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:21.364+0000 7fec7ad63700 1 -- 172.21.15.134:0/3601601778 wait complete. 2023-11-16T01:15:21.366 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:22.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:19.994697+0000 mgr.smithi133.xdyttl (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:22.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:21.362692+0000 mon.smithi133 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.134:0/3601601778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:22.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:22.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:24.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.024+0000 7fe11eb92700 1 -- 172.21.15.134:0/3095882839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 msgr2=0x7fe1180f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:24.368 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.024+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3095882839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1180f4b60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe10c009fb0 tx=0x7fe10c009b30).stop 2023-11-16T01:15:24.369 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.024+0000 7fe11eb92700 1 -- 172.21.15.134:0/3095882839 shutdown_connections 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.024+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3095882839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1180f4b60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.024+0000 7fe11eb92700 1 -- 172.21.15.134:0/3095882839 >> 172.21.15.134:0/3095882839 conn(0x7fe1180f0260 msgr2=0x7fe1180f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 -- 172.21.15.134:0/3095882839 shutdown_connections 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 -- 172.21.15.134:0/3095882839 wait complete. 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 Processor -- start 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 -- start start 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.025+0000 7fe11eb92700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe118106bc0 con 0x7fe1180f4740 2023-11-16T01:15:24.372 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11c92e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11c92e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:33308/0 (socket says 172.21.15.134:33308) 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11c92e700 1 -- 172.21.15.134:0/3284487207 learned_addr learned my addr 172.21.15.134:0/3284487207 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11c92e700 1 -- 172.21.15.134:0/3284487207 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c005040 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11c92e700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe10c013510 tx=0x7fe10c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c009320 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe118106dc0 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe10c013f00 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c014d90 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.026+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe118107250 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.027+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118045ad0 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.027+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe10c0095f0 con 0x7fe1180f4740 2023-11-16T01:15:24.373 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.027+0000 7fe1157fa700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 0x7fe108032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:24.374 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.027+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe10c041000 con 0x7fe1180f4740 2023-11-16T01:15:24.374 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:21.995009+0000 mgr.smithi133.xdyttl (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:24.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.028+0000 7fe117fff700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 0x7fe108032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:24.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.028+0000 7fe117fff700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 0x7fe108032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe1040096a0 tx=0x7fe104006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:24.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.032+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe10c014420 con 0x7fe1180f4740 2023-11-16T01:15:24.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.234+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe118108100 con 0x7fe1180f4740 2023-11-16T01:15:24.375 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.235+0000 7fe1157fa700 1 -- 172.21.15.134:0/3284487207 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe10c010230 con 0x7fe1180f4740 2023-11-16T01:15:24.376 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:24.376 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 msgr2=0x7fe108032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 0x7fe108032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe1040096a0 tx=0x7fe104006b40).stop 2023-11-16T01:15:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 msgr2=0x7fe1181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe10c013510 tx=0x7fe10c00b040).stop 2023-11-16T01:15:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 shutdown_connections 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fe10802ff60 0x7fe108032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 --2- 172.21.15.134:0/3284487207 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1180f4740 0x7fe1181066d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 >> 172.21.15.134:0/3284487207 conn(0x7fe1180f0260 msgr2=0x7fe1180f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 shutdown_connections 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:24.237+0000 7fe11eb92700 1 -- 172.21.15.134:0/3284487207 wait complete. 2023-11-16T01:15:27.084 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:27.085 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:24 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:24.235608+0000 mon.smithi133 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.134:0/3284487207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:27.085 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:23.995351+0000 mgr.smithi133.xdyttl (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:28.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:28.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:28.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:25.995701+0000 mgr.smithi133.xdyttl (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:29.289 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- 172.21.15.134:0/1188956196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 msgr2=0x7fef100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:29.289 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 --2- 172.21.15.134:0/1188956196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef100f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fef00009fb0 tx=0x7fef00009b30).stop 2023-11-16T01:15:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- 172.21.15.134:0/1188956196 shutdown_connections 2023-11-16T01:15:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 --2- 172.21.15.134:0/1188956196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef100f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- 172.21.15.134:0/1188956196 >> 172.21.15.134:0/1188956196 conn(0x7fef100f0b70 msgr2=0x7fef100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- 172.21.15.134:0/1188956196 shutdown_connections 2023-11-16T01:15:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- 172.21.15.134:0/1188956196 wait complete. 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 Processor -- start 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.289+0000 7fef1824a700 1 -- start start 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef1824a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef1824a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef10060220 con 0x7fef100f5410 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef15fe6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef15fe6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:45552/0 (socket says 172.21.15.134:45552) 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef15fe6700 1 -- 172.21.15.134:0/521199721 learned_addr learned my addr 172.21.15.134:0/521199721 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef15fe6700 1 -- 172.21.15.134:0/521199721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00005040 con 0x7fef100f5410 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef15fe6700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fef00013510 tx=0x7fef0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef00009320 con 0x7fef100f5410 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef00013f00 con 0x7fef100f5410 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef10060420 con 0x7fef100f5410 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.290+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef00014d90 con 0x7fef100f5410 2023-11-16T01:15:29.292 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.291+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef10109b00 con 0x7fef100f5410 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.291+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef000095f0 con 0x7fef100f5410 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.291+0000 7fef067fc700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 0x7feefc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.291+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef000406b0 con 0x7fef100f5410 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.292+0000 7fef157e5700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 0x7feefc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.292+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef10059100 con 0x7fef100f5410 2023-11-16T01:15:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.292+0000 7fef157e5700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 0x7feefc032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fef0c0096a0 tx=0x7fef0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:29.297 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.297+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef0001bb50 con 0x7fef100f5410 2023-11-16T01:15:29.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.501+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef10045ad0 con 0x7fef100f5410 2023-11-16T01:15:29.502 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.502+0000 7fef067fc700 1 -- 172.21.15.134:0/521199721 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef0001b740 con 0x7fef100f5410 2023-11-16T01:15:29.503 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:29.503 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:29.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 msgr2=0x7feefc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:29.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 0x7feefc032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fef0c0096a0 tx=0x7fef0c006b40).stop 2023-11-16T01:15:29.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 msgr2=0x7fef10109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fef00013510 tx=0x7fef0000b040).stop 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 shutdown_connections 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7feefc02ff60 0x7feefc032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 --2- 172.21.15.134:0/521199721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef100f5410 0x7fef10109450 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 >> 172.21.15.134:0/521199721 conn(0x7fef100f0b70 msgr2=0x7fef100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 shutdown_connections 2023-11-16T01:15:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:29.503+0000 7fef1824a700 1 -- 172.21.15.134:0/521199721 wait complete. 2023-11-16T01:15:29.506 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:27.996101+0000 mgr.smithi133.xdyttl (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:29.502168+0000 mon.smithi133 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.134:0/521199721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:30.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:30.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:30.077191+0000 mon.smithi133 (mon.0) 273 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:15:31.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:30.077982+0000 mon.smithi133 (mon.0) 274 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 -- 172.21.15.134:0/1632093988 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 msgr2=0x7f5c240f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1632093988 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c240f9170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5c0c009fb0 tx=0x7f5c0c009b30).stop 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 -- 172.21.15.134:0/1632093988 shutdown_connections 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1632093988 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c240f9170 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 -- 172.21.15.134:0/1632093988 >> 172.21.15.134:0/1632093988 conn(0x7f5c240f0b70 msgr2=0x7f5c240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 -- 172.21.15.134:0/1632093988 shutdown_connections 2023-11-16T01:15:32.185 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.184+0000 7f5c290d8700 1 -- 172.21.15.134:0/1632093988 wait complete. 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c290d8700 1 Processor -- start 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c290d8700 1 -- start start 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c290d8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c290d8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c24106b90 con 0x7f5c24060200 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c22d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c22d9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:45566/0 (socket says 172.21.15.134:45566) 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.185+0000 7f5c22d9d700 1 -- 172.21.15.134:0/1662022097 learned_addr learned my addr 172.21.15.134:0/1662022097 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c22d9d700 1 -- 172.21.15.134:0/1662022097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c005040 con 0x7f5c24060200 2023-11-16T01:15:32.187 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c22d9d700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5c0c013510 tx=0x7f5c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:32.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c0c009320 con 0x7f5c24060200 2023-11-16T01:15:32.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c0c013f00 con 0x7f5c24060200 2023-11-16T01:15:32.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c0c014dc0 con 0x7f5c24060200 2023-11-16T01:15:32.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c24106d90 con 0x7f5c24060200 2023-11-16T01:15:32.188 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.186+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c24107220 con 0x7f5c24060200 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.187+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c0c0095f0 con 0x7f5c24060200 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.187+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c24045ad0 con 0x7f5c24060200 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.187+0000 7f5c1b7fe700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 0x7f5c100324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.187+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c0c041070 con 0x7f5c24060200 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.187+0000 7f5c2259c700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 0x7f5c100324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:32.189 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.188+0000 7f5c2259c700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 0x7f5c100324c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5c140096a0 tx=0x7f5c14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:32.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.192+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c0c06d2b0 con 0x7f5c24060200 2023-11-16T01:15:32.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:29.996499+0000 mgr.smithi133.xdyttl (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:32.396 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.395+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c241080d0 con 0x7f5c24060200 2023-11-16T01:15:32.397 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.396+0000 7f5c1b7fe700 1 -- 172.21.15.134:0/1662022097 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c0c01d7f0 con 0x7f5c24060200 2023-11-16T01:15:32.397 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:32.397 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 msgr2=0x7f5c100324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 0x7f5c100324c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5c140096a0 tx=0x7f5c14006b40).stop 2023-11-16T01:15:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 msgr2=0x7f5c241066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5c0c013510 tx=0x7f5c0c00b040).stop 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 shutdown_connections 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f5c10030000 0x7f5c100324c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 --2- 172.21.15.134:0/1662022097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c24060200 0x7f5c241066a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 >> 172.21.15.134:0/1662022097 conn(0x7f5c240f0b70 msgr2=0x7f5c240f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 shutdown_connections 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:32.398+0000 7f5c290d8700 1 -- 172.21.15.134:0/1662022097 wait complete. 2023-11-16T01:15:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:33.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:32 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:32.396742+0000 mon.smithi133 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.134:0/1662022097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:33.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:33.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:34.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:31.996774+0000 mgr.smithi133.xdyttl (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 -- 172.21.15.134:0/985705827 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 msgr2=0x7f22680f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 --2- 172.21.15.134:0/985705827 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f22680f8f20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f2258009fb0 tx=0x7f2258009b30).stop 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 -- 172.21.15.134:0/985705827 shutdown_connections 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 --2- 172.21.15.134:0/985705827 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f22680f8f20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 -- 172.21.15.134:0/985705827 >> 172.21.15.134:0/985705827 conn(0x7f22680f09a0 msgr2=0x7f22680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 -- 172.21.15.134:0/985705827 shutdown_connections 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 -- 172.21.15.134:0/985705827 wait complete. 2023-11-16T01:15:35.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.027+0000 7f226d46d700 1 Processor -- start 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f226d46d700 1 -- start start 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f226d46d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f226d46d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22681069f0 con 0x7f22680601e0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2267fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2267fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:45594/0 (socket says 172.21.15.134:45594) 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2267fff700 1 -- 172.21.15.134:0/380077764 learned_addr learned my addr 172.21.15.134:0/380077764 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2267fff700 1 -- 172.21.15.134:0/380077764 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2258005040 con 0x7f22680601e0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2267fff700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f2258013510 tx=0x7f225800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2258009320 con 0x7f22680601e0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2268106bf0 con 0x7f22680601e0 2023-11-16T01:15:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2258013f00 con 0x7f22680601e0 2023-11-16T01:15:35.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2258014d90 con 0x7f22680601e0 2023-11-16T01:15:35.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.028+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2268107080 con 0x7f22680601e0 2023-11-16T01:15:35.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.029+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2268045ad0 con 0x7f22680601e0 2023-11-16T01:15:35.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.029+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22580095f0 con 0x7f22680601e0 2023-11-16T01:15:35.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.030+0000 7f2264ff9700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 0x7f2250032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:35.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.030+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2258040fc0 con 0x7f22680601e0 2023-11-16T01:15:35.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.030+0000 7f22677fe700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 0x7f2250032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:35.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.030+0000 7f22677fe700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 0x7f2250032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:35.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.034+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2258014420 con 0x7f22680601e0 2023-11-16T01:15:35.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.243+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2268107ed0 con 0x7f22680601e0 2023-11-16T01:15:35.244 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.244+0000 7f2264ff9700 1 -- 172.21.15.134:0/380077764 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2258010230 con 0x7f22680601e0 2023-11-16T01:15:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:35.245 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:35.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.245+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 msgr2=0x7f2250032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:35.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.245+0000 7f226d46d700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 0x7f2250032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f225c0096a0 tx=0x7f225c006b40).stop 2023-11-16T01:15:35.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.245+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 msgr2=0x7f2268106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:35.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.245+0000 7f226d46d700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f2258013510 tx=0x7f225800b040).stop 2023-11-16T01:15:35.246 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 shutdown_connections 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f225002ff60 0x7f2250032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 --2- 172.21.15.134:0/380077764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f22680601e0 0x7f2268106500 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 >> 172.21.15.134:0/380077764 conn(0x7f22680f09a0 msgr2=0x7f22680f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 shutdown_connections 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:35.246+0000 7f226d46d700 1 -- 172.21.15.134:0/380077764 wait complete. 2023-11-16T01:15:35.247 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:36.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:33.997133+0000 mgr.smithi133.xdyttl (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:36.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:35.244410+0000 mon.smithi133 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.134:0/380077764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:36.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:36.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:37.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 -- 172.21.15.134:0/1288067540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 msgr2=0x7f4c200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:37.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1288067540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c200f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f4c10009fb0 tx=0x7f4c10009b30).stop 2023-11-16T01:15:37.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 -- 172.21.15.134:0/1288067540 shutdown_connections 2023-11-16T01:15:37.899 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1288067540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c200f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 -- 172.21.15.134:0/1288067540 >> 172.21.15.134:0/1288067540 conn(0x7f4c200f1c70 msgr2=0x7f4c200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 -- 172.21.15.134:0/1288067540 shutdown_connections 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 -- 172.21.15.134:0/1288067540 wait complete. 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.896+0000 7f4c27edd700 1 Processor -- start 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c27edd700 1 -- start start 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c27edd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c27edd700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20106bc0 con 0x7f4c200f6510 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c25c79700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c25c79700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:45612/0 (socket says 172.21.15.134:45612) 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c25c79700 1 -- 172.21.15.134:0/1702175520 learned_addr learned my addr 172.21.15.134:0/1702175520 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c25c79700 1 -- 172.21.15.134:0/1702175520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10005040 con 0x7f4c200f6510 2023-11-16T01:15:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.897+0000 7f4c25c79700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f4c10013510 tx=0x7f4c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.898+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c10009320 con 0x7f4c200f6510 2023-11-16T01:15:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.898+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c10013f00 con 0x7f4c200f6510 2023-11-16T01:15:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.898+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c20106dc0 con 0x7f4c200f6510 2023-11-16T01:15:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.898+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c10014d90 con 0x7f4c200f6510 2023-11-16T01:15:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.898+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c20107250 con 0x7f4c200f6510 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c100095f0 con 0x7f4c200f6510 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c20045ad0 con 0x7f4c200f6510 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c167fc700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 0x7f4c0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c10041000 con 0x7f4c200f6510 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c25478700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 0x7f4c0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:37.902 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.899+0000 7f4c25478700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 0x7f4c0c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4c1c0096a0 tx=0x7f4c1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:37.905 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:37.904+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c10014420 con 0x7f4c200f6510 2023-11-16T01:15:38.113 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.111+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c20108100 con 0x7f4c200f6510 2023-11-16T01:15:38.113 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.112+0000 7f4c167fc700 1 -- 172.21.15.134:0/1702175520 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c10010230 con 0x7f4c200f6510 2023-11-16T01:15:38.114 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:38.114 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:38.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 msgr2=0x7f4c0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:38.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 0x7f4c0c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4c1c0096a0 tx=0x7f4c1c006b40).stop 2023-11-16T01:15:38.115 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 msgr2=0x7f4c201066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f4c10013510 tx=0x7f4c1000b040).stop 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 shutdown_connections 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f4c0c02ff60 0x7f4c0c032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 --2- 172.21.15.134:0/1702175520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4c200f6510 0x7f4c201066d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 >> 172.21.15.134:0/1702175520 conn(0x7f4c200f1c70 msgr2=0x7f4c200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 shutdown_connections 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:38.114+0000 7f4c27edd700 1 -- 172.21.15.134:0/1702175520 wait complete. 2023-11-16T01:15:38.116 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:38.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:35.997503+0000 mgr.smithi133.xdyttl (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:39.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:38 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:38.113524+0000 mon.smithi133 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.134:0/1702175520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:39.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:39.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:40.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:37.997875+0000 mgr.smithi133.xdyttl (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- 172.21.15.134:0/2440166066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 msgr2=0x7f73d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 --2- 172.21.15.134:0/2440166066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d00f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f73c0009fb0 tx=0x7f73c0009b30).stop 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- 172.21.15.134:0/2440166066 shutdown_connections 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 --2- 172.21.15.134:0/2440166066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d00f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- 172.21.15.134:0/2440166066 >> 172.21.15.134:0/2440166066 conn(0x7f73d00f09a0 msgr2=0x7f73d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- 172.21.15.134:0/2440166066 shutdown_connections 2023-11-16T01:15:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- 172.21.15.134:0/2440166066 wait complete. 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 Processor -- start 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.721+0000 7f73d673c700 1 -- start start 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d673c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d673c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0060220 con 0x7f73d00f51a0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d573a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d573a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46314/0 (socket says 172.21.15.134:46314) 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d573a700 1 -- 172.21.15.134:0/4146452061 learned_addr learned my addr 172.21.15.134:0/4146452061 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d573a700 1 -- 172.21.15.134:0/4146452061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0005040 con 0x7f73d00f51a0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d573a700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0009320 con 0x7f73d00f51a0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0060420 con 0x7f73d00f51a0 2023-11-16T01:15:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73c0013f00 con 0x7f73d00f51a0 2023-11-16T01:15:40.726 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.722+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0014dc0 con 0x7f73d00f51a0 2023-11-16T01:15:40.726 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.723+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d0109880 con 0x7f73d00f51a0 2023-11-16T01:15:40.726 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.723+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73c001d3d0 con 0x7f73d00f51a0 2023-11-16T01:15:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.724+0000 7f73c5ffb700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 0x7f73bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.724+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73c0041100 con 0x7f73d00f51a0 2023-11-16T01:15:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.724+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0059100 con 0x7f73d00f51a0 2023-11-16T01:15:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.724+0000 7f73d4f39700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 0x7f73bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.724+0000 7f73d4f39700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 0x7f73bc032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f73cc0096a0 tx=0x7f73cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:40.730 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.728+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73c0012920 con 0x7f73d00f51a0 2023-11-16T01:15:40.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.932+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73d00fabe0 con 0x7f73d00f51a0 2023-11-16T01:15:40.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.933+0000 7f73c5ffb700 1 -- 172.21.15.134:0/4146452061 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f73c002b020 con 0x7f73d00f51a0 2023-11-16T01:15:40.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:40.936 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:40.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 msgr2=0x7f73bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:40.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 0x7f73bc032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f73cc0096a0 tx=0x7f73cc006b40).stop 2023-11-16T01:15:40.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 msgr2=0x7f73d01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:40.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).stop 2023-11-16T01:15:40.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 shutdown_connections 2023-11-16T01:15:40.938 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f73bc02ff60 0x7f73bc032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 --2- 172.21.15.134:0/4146452061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73d00f51a0 0x7f73d01091d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 >> 172.21.15.134:0/4146452061 conn(0x7f73d00f09a0 msgr2=0x7f73d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 shutdown_connections 2023-11-16T01:15:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:40.935+0000 7f73d673c700 1 -- 172.21.15.134:0/4146452061 wait complete. 2023-11-16T01:15:40.939 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:42.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:39.998159+0000 mgr.smithi133.xdyttl (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:42.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:40.934375+0000 mon.smithi133 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.134:0/4146452061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:42.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:42.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:43.470 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- 172.21.15.134:0/2607975127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 msgr2=0x7fae9c069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 --2- 172.21.15.134:0/2607975127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c069e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fae98009fb0 tx=0x7fae98009b30).stop 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- 172.21.15.134:0/2607975127 shutdown_connections 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 --2- 172.21.15.134:0/2607975127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c069e40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- 172.21.15.134:0/2607975127 >> 172.21.15.134:0/2607975127 conn(0x7fae9c0f1c70 msgr2=0x7fae9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- 172.21.15.134:0/2607975127 shutdown_connections 2023-11-16T01:15:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- 172.21.15.134:0/2607975127 wait complete. 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 Processor -- start 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.468+0000 7faea3594700 1 -- start start 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea3594700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea3594700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c108de0 con 0x7fae9c06b940 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea1330700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea1330700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46342/0 (socket says 172.21.15.134:46342) 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea1330700 1 -- 172.21.15.134:0/3069076121 learned_addr learned my addr 172.21.15.134:0/3069076121 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea1330700 1 -- 172.21.15.134:0/3069076121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98005040 con 0x7fae9c06b940 2023-11-16T01:15:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea1330700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fae98013510 tx=0x7fae9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:43.473 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae98009320 con 0x7fae9c06b940 2023-11-16T01:15:43.473 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae98013f00 con 0x7fae9c06b940 2023-11-16T01:15:43.473 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c108fe0 con 0x7fae9c06b940 2023-11-16T01:15:43.474 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae98014dc0 con 0x7fae9c06b940 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.469+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c109470 con 0x7fae9c06b940 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.470+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae98014420 con 0x7fae9c06b940 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.470+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c045ad0 con 0x7fae9c06b940 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.470+0000 7fae91ffb700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 0x7fae88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.470+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae98041030 con 0x7fae9c06b940 2023-11-16T01:15:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.471+0000 7faea0b2f700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 0x7fae88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:43.476 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.471+0000 7faea0b2f700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 0x7fae88032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fae8c0096a0 tx=0x7fae8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:43.476 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.475+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae980095f0 con 0x7fae9c06b940 2023-11-16T01:15:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.685+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae9c10a320 con 0x7fae9c06b940 2023-11-16T01:15:43.687 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.686+0000 7fae91ffb700 1 -- 172.21.15.134:0/3069076121 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fae9801cb10 con 0x7fae9c06b940 2023-11-16T01:15:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:43.688 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:43.689 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.687+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 msgr2=0x7fae88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:43.689 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.687+0000 7faea3594700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 0x7fae88032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fae8c0096a0 tx=0x7fae8c006b40).stop 2023-11-16T01:15:43.689 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.687+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 msgr2=0x7fae9c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.687+0000 7faea3594700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fae98013510 tx=0x7fae9800b040).stop 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 shutdown_connections 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fae8802ff60 0x7fae88032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 --2- 172.21.15.134:0/3069076121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae9c06b940 0x7fae9c1088f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 >> 172.21.15.134:0/3069076121 conn(0x7fae9c0f1c70 msgr2=0x7fae9c0f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 shutdown_connections 2023-11-16T01:15:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:43.688+0000 7faea3594700 1 -- 172.21.15.134:0/3069076121 wait complete. 2023-11-16T01:15:43.691 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:41.998438+0000 mgr.smithi133.xdyttl (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:43.687131+0000 mon.smithi133 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.134:0/3069076121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:45.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:45.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 -- 172.21.15.134:0/3081034611 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 msgr2=0x7f7d680f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/3081034611 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680f5df0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f7d5c009fb0 tx=0x7f7d5c009b30).stop 2023-11-16T01:15:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 -- 172.21.15.134:0/3081034611 shutdown_connections 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/3081034611 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680f5df0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 -- 172.21.15.134:0/3081034611 >> 172.21.15.134:0/3081034611 conn(0x7f7d680f02e0 msgr2=0x7f7d680f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.214+0000 7f7d6d57d700 1 -- 172.21.15.134:0/3081034611 shutdown_connections 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.215+0000 7f7d6d57d700 1 -- 172.21.15.134:0/3081034611 wait complete. 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.215+0000 7f7d6d57d700 1 Processor -- start 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.215+0000 7f7d6d57d700 1 -- start start 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.215+0000 7f7d6d57d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.215+0000 7f7d6d57d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d680fb810 con 0x7f7d680f59d0 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d66ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d66ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46362/0 (socket says 172.21.15.134:46362) 2023-11-16T01:15:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d66ffd700 1 -- 172.21.15.134:0/2921070302 learned_addr learned my addr 172.21.15.134:0/2921070302 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d66ffd700 1 -- 172.21.15.134:0/2921070302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c005040 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d66ffd700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f7d5c013510 tx=0x7f7d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c009320 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d5c013f00 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d680fb9b0 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c014d90 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.216+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d680fbe40 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.217+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d5c0095f0 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.217+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68059100 con 0x7f7d680f59d0 2023-11-16T01:15:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.217+0000 7f7d4f7fe700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 0x7f7d50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.217+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d5c040fe0 con 0x7f7d680f59d0 2023-11-16T01:15:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.217+0000 7f7d667fc700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 0x7f7d50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.218+0000 7f7d667fc700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 0x7f7d50032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7d580096a0 tx=0x7f7d58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:46.223 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.222+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d5c01d7f0 con 0x7f7d680f59d0 2023-11-16T01:15:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:43.998736+0000 mgr.smithi133.xdyttl (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:46.429 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.428+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d680fae50 con 0x7f7d680f59d0 2023-11-16T01:15:46.430 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.428+0000 7f7d4f7fe700 1 -- 172.21.15.134:0/2921070302 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d5c010230 con 0x7f7d680f59d0 2023-11-16T01:15:46.430 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:46.430 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:46.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 msgr2=0x7f7d50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 0x7f7d50032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7d580096a0 tx=0x7f7d58006b40).stop 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 msgr2=0x7f7d680fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f7d5c013510 tx=0x7f7d5c00b040).stop 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 shutdown_connections 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7d5002ff60 0x7f7d50032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 --2- 172.21.15.134:0/2921070302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d680f59d0 0x7f7d680fd390 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 >> 172.21.15.134:0/2921070302 conn(0x7f7d680f02e0 msgr2=0x7f7d680f4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 shutdown_connections 2023-11-16T01:15:46.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:46.430+0000 7f7d6d57d700 1 -- 172.21.15.134:0/2921070302 wait complete. 2023-11-16T01:15:46.433 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:47.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:46 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:46.429821+0000 mon.smithi133 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.134:0/2921070302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:47.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:47.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:48.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:45.999076+0000 mgr.smithi133.xdyttl (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:49.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 -- 172.21.15.134:0/2434642289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 msgr2=0x7f838c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:49.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 --2- 172.21.15.134:0/2434642289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c0f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f837c009fb0 tx=0x7f837c009b30).stop 2023-11-16T01:15:49.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 -- 172.21.15.134:0/2434642289 shutdown_connections 2023-11-16T01:15:49.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 --2- 172.21.15.134:0/2434642289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c0f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:49.037 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 -- 172.21.15.134:0/2434642289 >> 172.21.15.134:0/2434642289 conn(0x7f838c0f1c70 msgr2=0x7f838c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 -- 172.21.15.134:0/2434642289 shutdown_connections 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.034+0000 7f8393471700 1 -- 172.21.15.134:0/2434642289 wait complete. 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f8393471700 1 Processor -- start 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f8393471700 1 -- start start 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f8393471700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f8393471700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c106c20 con 0x7f838c0f6510 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f839120d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f839120d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:46370/0 (socket says 172.21.15.134:46370) 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.035+0000 7f839120d700 1 -- 172.21.15.134:0/1879946639 learned_addr learned my addr 172.21.15.134:0/1879946639 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:49.038 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f839120d700 1 -- 172.21.15.134:0/1879946639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c005040 con 0x7f838c0f6510 2023-11-16T01:15:49.039 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f839120d700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f837c013510 tx=0x7f837c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f837c009320 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c106e20 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f837c013f00 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f837c014d90 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.036+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c1072b0 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.037+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c045ad0 con 0x7f838c0f6510 2023-11-16T01:15:49.040 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.037+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f837c0095f0 con 0x7f838c0f6510 2023-11-16T01:15:49.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.037+0000 7f8381ffb700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 0x7f8378032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:49.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.037+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f837c040fc0 con 0x7f838c0f6510 2023-11-16T01:15:49.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.038+0000 7f8390a0c700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 0x7f8378032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:49.041 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.038+0000 7f8390a0c700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 0x7f8378032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f83880096a0 tx=0x7f8388006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:49.043 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.042+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f837c014420 con 0x7f838c0f6510 2023-11-16T01:15:49.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.245+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f838c0fbf60 con 0x7f838c0f6510 2023-11-16T01:15:49.249 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.246+0000 7f8381ffb700 1 -- 172.21.15.134:0/1879946639 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f837c010230 con 0x7f838c0f6510 2023-11-16T01:15:49.252 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:49.253 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:49.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 msgr2=0x7f8378032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:49.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 0x7f8378032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f83880096a0 tx=0x7f8388006b40).stop 2023-11-16T01:15:49.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 msgr2=0x7f838c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:49.258 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f837c013510 tx=0x7f837c00b040).stop 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 shutdown_connections 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f837802ff60 0x7f8378032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 --2- 172.21.15.134:0/1879946639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f838c0f6510 0x7f838c106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 >> 172.21.15.134:0/1879946639 conn(0x7f838c0f1c70 msgr2=0x7f838c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 shutdown_connections 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:49.247+0000 7f8393471700 1 -- 172.21.15.134:0/1879946639 wait complete. 2023-11-16T01:15:49.259 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:47.999436+0000 mgr.smithi133.xdyttl (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:49.247325+0000 mon.smithi133 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.134:0/1879946639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:50.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:50.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- 172.21.15.134:0/2868066876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 msgr2=0x7f13980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 --2- 172.21.15.134:0/2868066876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13980f66c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f138c009fb0 tx=0x7f138c009b30).stop 2023-11-16T01:15:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- 172.21.15.134:0/2868066876 shutdown_connections 2023-11-16T01:15:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 --2- 172.21.15.134:0/2868066876 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13980f66c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:51.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- 172.21.15.134:0/2868066876 >> 172.21.15.134:0/2868066876 conn(0x7f13980f1aa0 msgr2=0x7f13980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:51.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- 172.21.15.134:0/2868066876 shutdown_connections 2023-11-16T01:15:51.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- 172.21.15.134:0/2868066876 wait complete. 2023-11-16T01:15:51.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 Processor -- start 2023-11-16T01:15:51.887 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.884+0000 7f139ded4700 1 -- start start 2023-11-16T01:15:51.889 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ded4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:51.889 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ded4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13981069a0 con 0x7f13980f62a0 2023-11-16T01:15:51.889 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ced2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:51.889 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ced2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49868/0 (socket says 172.21.15.134:49868) 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ced2700 1 -- 172.21.15.134:0/1741105893 learned_addr learned my addr 172.21.15.134:0/1741105893 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.885+0000 7f139ced2700 1 -- 172.21.15.134:0/1741105893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c005040 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f139ced2700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f138c013510 tx=0x7f138c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138c009320 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f138c013f00 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1398106ba0 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138c014dc0 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1398107030 con 0x7f13980f62a0 2023-11-16T01:15:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.886+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f138c01d3d0 con 0x7f13980f62a0 2023-11-16T01:15:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.887+0000 7f13957fa700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 0x7f1388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.887+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f138c0410b0 con 0x7f13980f62a0 2023-11-16T01:15:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.887+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1398045ad0 con 0x7f13980f62a0 2023-11-16T01:15:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.887+0000 7f1397fff700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 0x7f1388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.887+0000 7f1397fff700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 0x7f1388032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f13840096a0 tx=0x7f1384006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:51.893 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:51.891+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f138c012900 con 0x7f13980f62a0 2023-11-16T01:15:52.096 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.095+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1398107d50 con 0x7f13980f62a0 2023-11-16T01:15:52.098 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.095+0000 7f13957fa700 1 -- 172.21.15.134:0/1741105893 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f138c02b020 con 0x7f13980f62a0 2023-11-16T01:15:52.099 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:52.099 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:52.099 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 msgr2=0x7f1388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 0x7f1388032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f13840096a0 tx=0x7f1384006b40).stop 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 msgr2=0x7f13981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f138c013510 tx=0x7f138c00b040).stop 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 shutdown_connections 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f138802ff60 0x7f1388032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 --2- 172.21.15.134:0/1741105893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f13980f62a0 0x7f13981064b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 >> 172.21.15.134:0/1741105893 conn(0x7f13980f1aa0 msgr2=0x7f13980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 shutdown_connections 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:52.097+0000 7f139ded4700 1 -- 172.21.15.134:0/1741105893 wait complete. 2023-11-16T01:15:52.100 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:52.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:49.999691+0000 mgr.smithi133.xdyttl (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:53.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:52 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:52.096969+0000 mon.smithi133 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.134:0/1741105893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:53.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:53.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:54.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:52.000057+0000 mgr.smithi133.xdyttl (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- 172.21.15.134:0/671657017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 msgr2=0x7f81940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 --2- 172.21.15.134:0/671657017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f81940f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f8188009fb0 tx=0x7f8188009b30).stop 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- 172.21.15.134:0/671657017 shutdown_connections 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 --2- 172.21.15.134:0/671657017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f81940f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- 172.21.15.134:0/671657017 >> 172.21.15.134:0/671657017 conn(0x7f81940f0b70 msgr2=0x7f81940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- 172.21.15.134:0/671657017 shutdown_connections 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- 172.21.15.134:0/671657017 wait complete. 2023-11-16T01:15:54.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 Processor -- start 2023-11-16T01:15:54.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.748+0000 7f819ac5a700 1 -- start start 2023-11-16T01:15:54.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f819ac5a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:54.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f819ac5a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8194107240 con 0x7f81940f5410 2023-11-16T01:15:54.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81989f6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:54.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81989f6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49890/0 (socket says 172.21.15.134:49890) 2023-11-16T01:15:54.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81989f6700 1 -- 172.21.15.134:0/432090841 learned_addr learned my addr 172.21.15.134:0/432090841 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:54.752 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81989f6700 1 -- 172.21.15.134:0/432090841 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188005040 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81989f6700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8188013510 tx=0x7f818800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8188009320 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8194060220 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8188013f00 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8188014d90 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.749+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819410bd40 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.750+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81880095f0 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.750+0000 7f81917fa700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.750+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8194045ad0 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.750+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8188041000 con 0x7f81940f5410 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.751+0000 7f8193fff700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:54.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.751+0000 7f8193fff700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 0x7f8184032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f81800096a0 tx=0x7f8180006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:54.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.755+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8188014420 con 0x7f81940f5410 2023-11-16T01:15:54.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.958+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81940faf50 con 0x7f81940f5410 2023-11-16T01:15:54.960 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.958+0000 7f81917fa700 1 -- 172.21.15.134:0/432090841 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f818802b020 con 0x7f81940f5410 2023-11-16T01:15:54.960 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:54.960 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:54.961 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 msgr2=0x7f8184032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 0x7f8184032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f81800096a0 tx=0x7f8180006b40).stop 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 msgr2=0x7f819410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8188013510 tx=0x7f818800b040).stop 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 shutdown_connections 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f818402ff60 0x7f8184032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 --2- 172.21.15.134:0/432090841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f81940f5410 0x7f819410b850 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 >> 172.21.15.134:0/432090841 conn(0x7f81940f0b70 msgr2=0x7f81940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 shutdown_connections 2023-11-16T01:15:54.962 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:54.960+0000 7f819ac5a700 1 -- 172.21.15.134:0/432090841 wait complete. 2023-11-16T01:15:54.963 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:56.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:54.000350+0000 mgr.smithi133.xdyttl (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:56.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:54.959931+0000 mon.smithi133 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.134:0/432090841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:15:56.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:56.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:57.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:15:55.069845+0000 mgr.smithi133.xdyttl (mgr.14162) 229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi133 ... 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Verifying port 9100 ... 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 1 from systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: systemctl: stderr Job for ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133.service failed because a timeout was exceeded. 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: systemctl: stderr See "systemctl status ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133.service" and "journalctl -xe" for details. 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:15:57.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: main() 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: r = ctx.func(ctx) 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: return func(ctx) 2023-11-16T01:15:57.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ports=daemon_ports) 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: RuntimeError: Failed command: systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: yield (conn, connr) 2023-11-16T01:15:57.283 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: code, '\n'.join(err))) 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi133 ... 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Verifying port 9100 ... 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Non-zero exit code 1 from systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: systemctl: stderr Job for ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133.service failed because a timeout was exceeded. 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: systemctl: stderr See "systemctl status ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133.service" and "journalctl -xe" for details. 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: Traceback (most recent call last): 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: main() 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: r = ctx.func(ctx) 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-16T01:15:57.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: return func(ctx) 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: ports=daemon_ports) 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: File "/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: RuntimeError: Failed command: systemctl start ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@node-exporter.smithi133 2023-11-16T01:15:57.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:56 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cephadm 2023-11-16T01:15:55.075383+0000 mgr.smithi133.xdyttl (mgr.14162) 230 : cephadm [INF] Deploying daemon prometheus.smithi133 on smithi133 2023-11-16T01:15:57.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 -- 172.21.15.134:0/3121707057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 msgr2=0x7efde00fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:57.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 --2- 172.21.15.134:0/3121707057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde00fb4f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7efdd0009fb0 tx=0x7efdd0009b30).stop 2023-11-16T01:15:57.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 -- 172.21.15.134:0/3121707057 shutdown_connections 2023-11-16T01:15:57.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 --2- 172.21.15.134:0/3121707057 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde00fb4f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:57.655 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 -- 172.21.15.134:0/3121707057 >> 172.21.15.134:0/3121707057 conn(0x7efde00f0b70 msgr2=0x7efde00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 -- 172.21.15.134:0/3121707057 shutdown_connections 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 -- 172.21.15.134:0/3121707057 wait complete. 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.652+0000 7efde68a6700 1 Processor -- start 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efde68a6700 1 -- start start 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efde68a6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efde68a6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde0108dd0 con 0x7efde0060280 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efddffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efddffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:49912/0 (socket says 172.21.15.134:49912) 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efddffff700 1 -- 172.21.15.134:0/2773121681 learned_addr learned my addr 172.21.15.134:0/2773121681 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efddffff700 1 -- 172.21.15.134:0/2773121681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0005040 con 0x7efde0060280 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.653+0000 7efddffff700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7efdd0013510 tx=0x7efdd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdd0009320 con 0x7efde0060280 2023-11-16T01:15:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdd0013f00 con 0x7efde0060280 2023-11-16T01:15:57.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde0108fd0 con 0x7efde0060280 2023-11-16T01:15:57.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdd0014dc0 con 0x7efde0060280 2023-11-16T01:15:57.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde0109460 con 0x7efde0060280 2023-11-16T01:15:57.657 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdd001d3d0 con 0x7efde0060280 2023-11-16T01:15:57.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0045ad0 con 0x7efde0060280 2023-11-16T01:15:57.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.654+0000 7efddcff9700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:15:57.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.655+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdd0041410 con 0x7efde0060280 2023-11-16T01:15:57.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.655+0000 7efddf7fe700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:15:57.658 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.655+0000 7efddf7fe700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 0x7efdcc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7efdc80096a0 tx=0x7efdc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:15:57.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.659+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdd0012510 con 0x7efde0060280 2023-11-16T01:15:57.863 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.862+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efde010a310 con 0x7efde0060280 2023-11-16T01:15:57.864 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.863+0000 7efddcff9700 1 -- 172.21.15.134:0/2773121681 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efdd0012330 con 0x7efde0060280 2023-11-16T01:15:57.864 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:15:57.864 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:15:57.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 msgr2=0x7efdcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:57.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 0x7efdcc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7efdc80096a0 tx=0x7efdc8006b40).stop 2023-11-16T01:15:57.866 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 msgr2=0x7efde01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7efdd0013510 tx=0x7efdd000b040).stop 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 shutdown_connections 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 --2- 172.21.15.134:0/2773121681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efde0060280 0x7efde01088e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 >> 172.21.15.134:0/2773121681 conn(0x7efde00f0b70 msgr2=0x7efde00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 shutdown_connections 2023-11-16T01:15:57.867 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:15:57.864+0000 7efde68a6700 1 -- 172.21.15.134:0/2773121681 wait complete. 2023-11-16T01:15:57.868 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:15:58.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:56.000620+0000 mgr.smithi133.xdyttl (mgr.14162) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:15:59.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:15:59.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:15:59.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:58 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:15:57.864030+0000 mon.smithi133 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.134:0/2773121681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:00.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:15:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:15:58.000960+0000 mgr.smithi133.xdyttl (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- 172.21.15.134:0/1056582589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 msgr2=0x7f32480f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 --2- 172.21.15.134:0/1056582589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32480f9170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f323c009fb0 tx=0x7f323c009b30).stop 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- 172.21.15.134:0/1056582589 shutdown_connections 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 --2- 172.21.15.134:0/1056582589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32480f9170 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- 172.21.15.134:0/1056582589 >> 172.21.15.134:0/1056582589 conn(0x7f32480f0b70 msgr2=0x7f32480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- 172.21.15.134:0/1056582589 shutdown_connections 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- 172.21.15.134:0/1056582589 wait complete. 2023-11-16T01:16:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 Processor -- start 2023-11-16T01:16:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.429+0000 7f324f16f700 1 -- start start 2023-11-16T01:16:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324f16f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324f16f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3248106b90 con 0x7f3248060200 2023-11-16T01:16:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324cf0b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324cf0b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34856/0 (socket says 172.21.15.134:34856) 2023-11-16T01:16:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324cf0b700 1 -- 172.21.15.134:0/18324185 learned_addr learned my addr 172.21.15.134:0/18324185 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324cf0b700 1 -- 172.21.15.134:0/18324185 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c005040 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324cf0b700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f323c013510 tx=0x7f323c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f323c009320 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3248106d90 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f323c013f00 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f323c014dc0 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.430+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3248107220 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.431+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f323c01d3d0 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.431+0000 7f32457fa700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.431+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f323c041110 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.431+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3248045ad0 con 0x7f3248060200 2023-11-16T01:16:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.432+0000 7f3247fff700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.432+0000 7f3247fff700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 0x7f3238032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f32340096a0 tx=0x7f3234006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:00.437 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.437+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f323c012920 con 0x7f3248060200 2023-11-16T01:16:00.647 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.641+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3248107f40 con 0x7f3248060200 2023-11-16T01:16:00.647 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.643+0000 7f32457fa700 1 -- 172.21.15.134:0/18324185 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f323c02b020 con 0x7f3248060200 2023-11-16T01:16:00.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:00.647 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:00.648 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 msgr2=0x7f3238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 0x7f3238032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f32340096a0 tx=0x7f3234006b40).stop 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 msgr2=0x7f32481066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f323c013510 tx=0x7f323c00b040).stop 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 shutdown_connections 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 --2- 172.21.15.134:0/18324185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248060200 0x7f32481066a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:00.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 >> 172.21.15.134:0/18324185 conn(0x7f32480f0b70 msgr2=0x7f32480f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:00.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 shutdown_connections 2023-11-16T01:16:00.650 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:00.644+0000 7f324f16f700 1 -- 172.21.15.134:0/18324185 wait complete. 2023-11-16T01:16:00.650 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:01.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:00.643429+0000 mon.smithi133 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.134:0/18324185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:02.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:02.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:02.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:00.001256+0000 mgr.smithi133.xdyttl (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/58691203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 msgr2=0x7fc29c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/58691203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fc28c009fb0 tx=0x7fc28c009b30).stop 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/58691203 shutdown_connections 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/58691203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/58691203 >> 172.21.15.134:0/58691203 conn(0x7fc29c0f1c70 msgr2=0x7fc29c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/58691203 shutdown_connections 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.204+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/58691203 wait complete. 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a3b3a700 1 Processor -- start 2023-11-16T01:16:03.206 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a3b3a700 1 -- start start 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a3b3a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a3b3a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c069e20 con 0x7fc29c0f6510 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a18d6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a18d6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34872/0 (socket says 172.21.15.134:34872) 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a18d6700 1 -- 172.21.15.134:0/1900691431 learned_addr learned my addr 172.21.15.134:0/1900691431 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.205+0000 7fc2a18d6700 1 -- 172.21.15.134:0/1900691431 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c005040 con 0x7fc29c0f6510 2023-11-16T01:16:03.207 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2a18d6700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc28c013510 tx=0x7fc28c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:03.208 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c009320 con 0x7fc29c0f6510 2023-11-16T01:16:03.208 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c069fc0 con 0x7fc29c0f6510 2023-11-16T01:16:03.208 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc28c013f00 con 0x7fc29c0f6510 2023-11-16T01:16:03.208 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c014d90 con 0x7fc29c0f6510 2023-11-16T01:16:03.208 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.206+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c06a450 con 0x7fc29c0f6510 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc28c0095f0 con 0x7fc29c0f6510 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c059100 con 0x7fc29c0f6510 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2927fc700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 0x7fc288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc28c041000 con 0x7fc29c0f6510 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2a10d5700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 0x7fc288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:03.209 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.207+0000 7fc2a10d5700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 0x7fc288032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc2980096a0 tx=0x7fc298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:03.213 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.211+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc28c014420 con 0x7fc29c0f6510 2023-11-16T01:16:03.421 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.418+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc29c0fbf60 con 0x7fc29c0f6510 2023-11-16T01:16:03.422 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.419+0000 7fc2927fc700 1 -- 172.21.15.134:0/1900691431 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc28c010230 con 0x7fc29c0f6510 2023-11-16T01:16:03.424 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:03.426 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:03.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 msgr2=0x7fc288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:03.432 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 0x7fc288032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc2980096a0 tx=0x7fc298006b40).stop 2023-11-16T01:16:03.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 msgr2=0x7fc29c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:03.433 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc28c013510 tx=0x7fc28c00b040).stop 2023-11-16T01:16:03.434 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 shutdown_connections 2023-11-16T01:16:03.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc28802ff60 0x7fc288032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:03.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 --2- 172.21.15.134:0/1900691431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc29c0f6510 0x7fc29c06b9a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:03.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 >> 172.21.15.134:0/1900691431 conn(0x7fc29c0f1c70 msgr2=0x7fc29c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:03.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 shutdown_connections 2023-11-16T01:16:03.436 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:03.420+0000 7fc2a3b3a700 1 -- 172.21.15.134:0/1900691431 wait complete. 2023-11-16T01:16:03.437 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:04.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:02.001531+0000 mgr.smithi133.xdyttl (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:04.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:03.419363+0000 mon.smithi133 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.134:0/1900691431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:04.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:04.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- 172.21.15.134:0/3102660248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 msgr2=0x7f2acc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 --2- 172.21.15.134:0/3102660248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc0f55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f2abc009fb0 tx=0x7f2abc009b30).stop 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- 172.21.15.134:0/3102660248 shutdown_connections 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 --2- 172.21.15.134:0/3102660248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc0f55c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- 172.21.15.134:0/3102660248 >> 172.21.15.134:0/3102660248 conn(0x7f2acc0f09a0 msgr2=0x7f2acc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- 172.21.15.134:0/3102660248 shutdown_connections 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- 172.21.15.134:0/3102660248 wait complete. 2023-11-16T01:16:05.984 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 Processor -- start 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- start start 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.982+0000 7f2acb59e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc060240 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2aca59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2aca59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34892/0 (socket says 172.21.15.134:34892) 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2aca59c700 1 -- 172.21.15.134:0/86120875 learned_addr learned my addr 172.21.15.134:0/86120875 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2aca59c700 1 -- 172.21.15.134:0/86120875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc005040 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2aca59c700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2abc013510 tx=0x7f2abc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2abc009320 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc060440 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2abc013f00 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2abc014dc0 con 0x7f2acc0f51a0 2023-11-16T01:16:05.985 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.983+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc109860 con 0x7f2acc0f51a0 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.984+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2abc014420 con 0x7f2acc0f51a0 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.984+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc045ad0 con 0x7f2acc0f51a0 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.984+0000 7f2abaffd700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 0x7f2ab4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.984+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2abc041110 con 0x7f2acc0f51a0 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.984+0000 7f2ac9d9b700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 0x7f2ab4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:05.987 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.985+0000 7f2ac9d9b700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 0x7f2ab4032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2ac00096a0 tx=0x7f2ac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:05.991 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:05.989+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2abc0095f0 con 0x7f2acc0f51a0 2023-11-16T01:16:06.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.195+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2acc10a080 con 0x7f2acc0f51a0 2023-11-16T01:16:06.196 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.195+0000 7f2abaffd700 1 -- 172.21.15.134:0/86120875 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2abc01cb10 con 0x7f2acc0f51a0 2023-11-16T01:16:06.196 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:06.196 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 msgr2=0x7f2ab4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 0x7f2ab4032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2ac00096a0 tx=0x7f2ac0006b40).stop 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 msgr2=0x7f2acc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2abc013510 tx=0x7f2abc00b040).stop 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 shutdown_connections 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ab402ffb0 0x7f2ab4032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:06.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 --2- 172.21.15.134:0/86120875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2acc0f51a0 0x7f2acc1091b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:06.199 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 >> 172.21.15.134:0/86120875 conn(0x7f2acc0f09a0 msgr2=0x7f2acc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:06.199 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 shutdown_connections 2023-11-16T01:16:06.199 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:06.197+0000 7f2acb59e700 1 -- 172.21.15.134:0/86120875 wait complete. 2023-11-16T01:16:06.199 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:06.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:04.001872+0000 mgr.smithi133.xdyttl (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:07.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:06 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:06.196006+0000 mon.smithi133 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.134:0/86120875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:07.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:07.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:08.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:06.002226+0000 mgr.smithi133.xdyttl (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:08.717 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 -- 172.21.15.134:0/2126263016 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 msgr2=0x7f92fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:08.717 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 --2- 172.21.15.134:0/2126263016 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc0f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f92e4009fb0 tx=0x7f92e4009b30).stop 2023-11-16T01:16:08.717 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 -- 172.21.15.134:0/2126263016 shutdown_connections 2023-11-16T01:16:08.717 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 --2- 172.21.15.134:0/2126263016 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc0f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 -- 172.21.15.134:0/2126263016 >> 172.21.15.134:0/2126263016 conn(0x7f92fc0f1c70 msgr2=0x7f92fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 -- 172.21.15.134:0/2126263016 shutdown_connections 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.716+0000 7f9300cb0700 1 -- 172.21.15.134:0/2126263016 wait complete. 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f9300cb0700 1 Processor -- start 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f9300cb0700 1 -- start start 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f9300cb0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:08.718 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f9300cb0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc106c20 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f92fa59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f92fa59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:34910/0 (socket says 172.21.15.134:34910) 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f92fa59c700 1 -- 172.21.15.134:0/1498060071 learned_addr learned my addr 172.21.15.134:0/1498060071 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f92fa59c700 1 -- 172.21.15.134:0/1498060071 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4005040 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.717+0000 7f92fa59c700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f92e4013510 tx=0x7f92e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92e4009320 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92e4013f00 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc106e20 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92e4014d90 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc1072b0 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92e40095f0 con 0x7f92fc0f6510 2023-11-16T01:16:08.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.718+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc045ad0 con 0x7f92fc0f6510 2023-11-16T01:16:08.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.719+0000 7f92f2ffd700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 0x7f92e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:08.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.719+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92e4041080 con 0x7f92fc0f6510 2023-11-16T01:16:08.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.719+0000 7f92f9d9b700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 0x7f92e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:08.721 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.719+0000 7f92f9d9b700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 0x7f92e8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f92ec0096a0 tx=0x7f92ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:08.724 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.723+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92e4014420 con 0x7f92fc0f6510 2023-11-16T01:16:08.931 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.930+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92fc0fbf60 con 0x7f92fc0f6510 2023-11-16T01:16:08.931 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.931+0000 7f92f2ffd700 1 -- 172.21.15.134:0/1498060071 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f92e4010230 con 0x7f92fc0f6510 2023-11-16T01:16:08.932 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:08.932 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:08.933 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 msgr2=0x7f92e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:08.933 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 0x7f92e8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f92ec0096a0 tx=0x7f92ec006b40).stop 2023-11-16T01:16:08.933 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 msgr2=0x7f92fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f92e4013510 tx=0x7f92e400b040).stop 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 shutdown_connections 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f92e802ff60 0x7f92e8032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 --2- 172.21.15.134:0/1498060071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92fc0f6510 0x7f92fc106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 >> 172.21.15.134:0/1498060071 conn(0x7f92fc0f1c70 msgr2=0x7f92fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 shutdown_connections 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:08.933+0000 7f9300cb0700 1 -- 172.21.15.134:0/1498060071 wait complete. 2023-11-16T01:16:08.934 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:08.002512+0000 mgr.smithi133.xdyttl (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:10.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:08.931818+0000 mon.smithi133 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.134:0/1498060071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:10.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:10.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/2336813664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 msgr2=0x7f2ebc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/2336813664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc0f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2eb8009fb0 tx=0x7f2eb8009b30).stop 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/2336813664 shutdown_connections 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/2336813664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc0f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/2336813664 >> 172.21.15.134:0/2336813664 conn(0x7f2ebc0f1c70 msgr2=0x7f2ebc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/2336813664 shutdown_connections 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.539+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/2336813664 wait complete. 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec3dd4700 1 Processor -- start 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec3dd4700 1 -- start start 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec3dd4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:11.541 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec3dd4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc106c20 con 0x7f2ebc0f6510 2023-11-16T01:16:11.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec1b70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:11.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec1b70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59164/0 (socket says 172.21.15.134:59164) 2023-11-16T01:16:11.542 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.540+0000 7f2ec1b70700 1 -- 172.21.15.134:0/3259852963 learned_addr learned my addr 172.21.15.134:0/3259852963 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:11.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2ec1b70700 1 -- 172.21.15.134:0/3259852963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8005040 con 0x7f2ebc0f6510 2023-11-16T01:16:11.543 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2ec1b70700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2eb8013510 tx=0x7f2eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb8009320 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc106e20 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2eb8013f00 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb8014d90 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.541+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc1072b0 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2eb8014420 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2eb27fc700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 0x7f2ea8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc045ad0 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2eb8040ff0 con 0x7f2ebc0f6510 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2ec136f700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 0x7f2ea8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:11.544 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.542+0000 7f2ec136f700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 0x7f2ea8032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f2eac0096a0 tx=0x7f2eac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:11.546 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.546+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2eb801db70 con 0x7f2ebc0f6510 2023-11-16T01:16:11.755 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.755+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ebc0fc050 con 0x7f2ebc0f6510 2023-11-16T01:16:11.756 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.755+0000 7f2eb27fc700 1 -- 172.21.15.134:0/3259852963 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2eb802c090 con 0x7f2ebc0f6510 2023-11-16T01:16:11.756 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:11.756 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:11.757 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 msgr2=0x7f2ea8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 0x7f2ea8032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f2eac0096a0 tx=0x7f2eac006b40).stop 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 msgr2=0x7f2ebc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2eb8013510 tx=0x7f2eb800b040).stop 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 shutdown_connections 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f2ea802ffb0 0x7f2ea8032470 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 --2- 172.21.15.134:0/3259852963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ebc0f6510 0x7f2ebc106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 >> 172.21.15.134:0/3259852963 conn(0x7f2ebc0f1c70 msgr2=0x7f2ebc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 shutdown_connections 2023-11-16T01:16:11.758 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:11.757+0000 7f2ec3dd4700 1 -- 172.21.15.134:0/3259852963 wait complete. 2023-11-16T01:16:11.759 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:12.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:10.002792+0000 mgr.smithi133.xdyttl (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:12.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:11.755964+0000 mon.smithi133 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.134:0/3259852963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:13.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:13.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:14.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:12.003159+0000 mgr.smithi133.xdyttl (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:14.383 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 -- 172.21.15.134:0/972605032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 msgr2=0x7f015c0f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 --2- 172.21.15.134:0/972605032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f45c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f014c009fb0 tx=0x7f014c009b30).stop 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 -- 172.21.15.134:0/972605032 shutdown_connections 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 --2- 172.21.15.134:0/972605032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f45c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 -- 172.21.15.134:0/972605032 >> 172.21.15.134:0/972605032 conn(0x7f015c0efe40 msgr2=0x7f015c0f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.382+0000 7f0161834700 1 -- 172.21.15.134:0/972605032 shutdown_connections 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.383+0000 7f0161834700 1 -- 172.21.15.134:0/972605032 wait complete. 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.383+0000 7f0161834700 1 Processor -- start 2023-11-16T01:16:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.383+0000 7f0161834700 1 -- start start 2023-11-16T01:16:14.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.383+0000 7f0161834700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:14.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.383+0000 7f0161834700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0f5c60 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f015affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f015affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59190/0 (socket says 172.21.15.134:59190) 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f015affd700 1 -- 172.21.15.134:0/1706914487 learned_addr learned my addr 172.21.15.134:0/1706914487 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f015affd700 1 -- 172.21.15.134:0/1706914487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c005040 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f015affd700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f014c013510 tx=0x7f014c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f014c009320 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c0f5e00 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f014c013f00 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f014c014dc0 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.384+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c0f6290 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.385+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f014c014420 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.385+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c059100 con 0x7f015c0f41a0 2023-11-16T01:16:14.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.385+0000 7f01437fe700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 0x7f0144032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:14.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.385+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f014c041050 con 0x7f015c0f41a0 2023-11-16T01:16:14.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.386+0000 7f015a7fc700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 0x7f0144032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:14.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.386+0000 7f015a7fc700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 0x7f0144032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f01500096a0 tx=0x7f0150006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:14.396 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.390+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f014c0095f0 con 0x7f015c0f41a0 2023-11-16T01:16:14.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.595+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f015c0fbf40 con 0x7f015c0f41a0 2023-11-16T01:16:14.596 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.596+0000 7f01437fe700 1 -- 172.21.15.134:0/1706914487 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f014c01cb10 con 0x7f015c0f41a0 2023-11-16T01:16:14.597 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:14.597 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:14.598 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 msgr2=0x7f0144032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 0x7f0144032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f01500096a0 tx=0x7f0150006b40).stop 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 msgr2=0x7f015c0f77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f014c013510 tx=0x7f014c00b040).stop 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 shutdown_connections 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f014402ff60 0x7f0144032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 --2- 172.21.15.134:0/1706914487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f015c0f41a0 0x7f015c0f77e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 >> 172.21.15.134:0/1706914487 conn(0x7f015c0efe40 msgr2=0x7f015c0f0950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 shutdown_connections 2023-11-16T01:16:14.599 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:14.598+0000 7f0161834700 1 -- 172.21.15.134:0/1706914487 wait complete. 2023-11-16T01:16:14.600 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:15.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:14 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:14.596722+0000 mon.smithi133 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.134:0/1706914487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:16.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:16.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:16.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:14.003441+0000 mgr.smithi133.xdyttl (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:17.283 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.282+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1231245092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 msgr2=0x7fced40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:17.283 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.282+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1231245092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced40f9150 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcec8009fb0 tx=0x7fcec8009b30).stop 2023-11-16T01:16:17.283 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1231245092 shutdown_connections 2023-11-16T01:16:17.283 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1231245092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced40f9150 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:17.284 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1231245092 >> 172.21.15.134:0/1231245092 conn(0x7fced40f0b70 msgr2=0x7fced40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:17.284 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1231245092 shutdown_connections 2023-11-16T01:16:17.284 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1231245092 wait complete. 2023-11-16T01:16:17.285 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 Processor -- start 2023-11-16T01:16:17.285 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- start start 2023-11-16T01:16:17.285 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:17.285 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fcedb3c1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4106ba0 con 0x7fced4060220 2023-11-16T01:16:17.285 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fced915d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fced915d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59202/0 (socket says 172.21.15.134:59202) 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.283+0000 7fced915d700 1 -- 172.21.15.134:0/1328957679 learned_addr learned my addr 172.21.15.134:0/1328957679 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fced915d700 1 -- 172.21.15.134:0/1328957679 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec8005040 con 0x7fced4060220 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fced915d700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fcec8013510 tx=0x7fcec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec8009320 con 0x7fced4060220 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4106da0 con 0x7fced4060220 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcec8013f00 con 0x7fced4060220 2023-11-16T01:16:17.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec8014dc0 con 0x7fced4060220 2023-11-16T01:16:17.287 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.284+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced4107230 con 0x7fced4060220 2023-11-16T01:16:17.287 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.285+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcec801d3d0 con 0x7fced4060220 2023-11-16T01:16:17.288 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.285+0000 7fcec5ffb700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 0x7fcec0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:17.288 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.285+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4045ad0 con 0x7fced4060220 2023-11-16T01:16:17.288 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.285+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcec80410c0 con 0x7fced4060220 2023-11-16T01:16:17.288 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.285+0000 7fced895c700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 0x7fcec0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:17.288 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.286+0000 7fced895c700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 0x7fcec0032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:17.291 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.290+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcec8012920 con 0x7fced4060220 2023-11-16T01:16:17.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.493+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fced4108080 con 0x7fced4060220 2023-11-16T01:16:17.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.493+0000 7fcec5ffb700 1 -- 172.21.15.134:0/1328957679 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcec8012330 con 0x7fced4060220 2023-11-16T01:16:17.495 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:17.495 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:17.496 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 msgr2=0x7fcec0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:17.496 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 0x7fcec0032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).stop 2023-11-16T01:16:17.496 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 msgr2=0x7fced41066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:17.496 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fcec8013510 tx=0x7fcec800b040).stop 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 shutdown_connections 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcec002ffb0 0x7fcec0032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 --2- 172.21.15.134:0/1328957679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fced4060220 0x7fced41066b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 >> 172.21.15.134:0/1328957679 conn(0x7fced40f0b70 msgr2=0x7fced40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 shutdown_connections 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:17.495+0000 7fcedb3c1700 1 -- 172.21.15.134:0/1328957679 wait complete. 2023-11-16T01:16:17.497 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:18.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:16.003734+0000 mgr.smithi133.xdyttl (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:18.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:17.493863+0000 mon.smithi133 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.134:0/1328957679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:18.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:18.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:20.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 -- 172.21.15.134:0/3995800048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 msgr2=0x7fa5380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/3995800048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa5380f55c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fa530009fb0 tx=0x7fa530009b30).stop 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 -- 172.21.15.134:0/3995800048 shutdown_connections 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/3995800048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa5380f55c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 -- 172.21.15.134:0/3995800048 >> 172.21.15.134:0/3995800048 conn(0x7fa5380f09a0 msgr2=0x7fa5380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 -- 172.21.15.134:0/3995800048 shutdown_connections 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.100+0000 7fa53e0ad700 1 -- 172.21.15.134:0/3995800048 wait complete. 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53e0ad700 1 Processor -- start 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53e0ad700 1 -- start start 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53e0ad700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53e0ad700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538104770 con 0x7fa5380f51a0 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53d0ab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:20.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53d0ab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40838/0 (socket says 172.21.15.134:40838) 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53d0ab700 1 -- 172.21.15.134:0/925922624 learned_addr learned my addr 172.21.15.134:0/925922624 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.101+0000 7fa53d0ab700 1 -- 172.21.15.134:0/925922624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530005040 con 0x7fa5380f51a0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa53d0ab700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa530013510 tx=0x7fa53000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa530009320 con 0x7fa5380f51a0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa538104970 con 0x7fa5380f51a0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa530013f00 con 0x7fa5380f51a0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa530014dc0 con 0x7fa5380f51a0 2023-11-16T01:16:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.102+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa538104e00 con 0x7fa5380f51a0 2023-11-16T01:16:20.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.103+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa530014420 con 0x7fa5380f51a0 2023-11-16T01:16:20.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.103+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa538045ad0 con 0x7fa5380f51a0 2023-11-16T01:16:20.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.103+0000 7fa52dffb700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 0x7fa524032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:20.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.103+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa530041030 con 0x7fa5380f51a0 2023-11-16T01:16:20.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.103+0000 7fa53c8aa700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 0x7fa524032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:20.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.104+0000 7fa53c8aa700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 0x7fa524032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa5280096a0 tx=0x7fa528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:20.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.108+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5300095f0 con 0x7fa5380f51a0 2023-11-16T01:16:20.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:18.004049+0000 mgr.smithi133.xdyttl (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.313+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5380fabe0 con 0x7fa5380f51a0 2023-11-16T01:16:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.314+0000 7fa52dffb700 1 -- 172.21.15.134:0/925922624 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa53001cb10 con 0x7fa5380f51a0 2023-11-16T01:16:20.315 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:20.315 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:20.317 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 msgr2=0x7fa524032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 0x7fa524032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa5280096a0 tx=0x7fa528006b40).stop 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 msgr2=0x7fa538104280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa530013510 tx=0x7fa53000b040).stop 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 shutdown_connections 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa52402ff60 0x7fa524032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 --2- 172.21.15.134:0/925922624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa5380f51a0 0x7fa538104280 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 >> 172.21.15.134:0/925922624 conn(0x7fa5380f09a0 msgr2=0x7fa53805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 shutdown_connections 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:20.315+0000 7fa53e0ad700 1 -- 172.21.15.134:0/925922624 wait complete. 2023-11-16T01:16:20.318 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:21.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:20 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:20.315238+0000 mon.smithi133 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.134:0/925922624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:21.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:21.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:22.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:20.004412+0000 mgr.smithi133.xdyttl (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1265732423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 msgr2=0x7f3d200f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1265732423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d200f5d70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3d14009fb0 tx=0x7f3d14009b30).stop 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1265732423 shutdown_connections 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1265732423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d200f5d70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1265732423 >> 172.21.15.134:0/1265732423 conn(0x7f3d200f0260 msgr2=0x7f3d200f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1265732423 shutdown_connections 2023-11-16T01:16:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.892+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1265732423 wait complete. 2023-11-16T01:16:22.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d27f3d700 1 Processor -- start 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d27f3d700 1 -- start start 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d27f3d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d27f3d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d200f5330 con 0x7f3d200f5950 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d25cd9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d25cd9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40846/0 (socket says 172.21.15.134:40846) 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d25cd9700 1 -- 172.21.15.134:0/1680200751 learned_addr learned my addr 172.21.15.134:0/1680200751 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d25cd9700 1 -- 172.21.15.134:0/1680200751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14005040 con 0x7f3d200f5950 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.893+0000 7f3d25cd9700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3d14013510 tx=0x7f3d1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d14009320 con 0x7f3d200f5950 2023-11-16T01:16:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d200f5530 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d14013f00 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d14014d90 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20109ac0 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.894+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d140095f0 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.895+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d20045ad0 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.895+0000 7f3d127fc700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.895+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d140410c0 con 0x7f3d200f5950 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.895+0000 7f3d254d8700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:22.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.895+0000 7f3d254d8700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 0x7f3d0c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3d1c0096a0 tx=0x7f3d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:22.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:22.899+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d1401d7f0 con 0x7f3d200f5950 2023-11-16T01:16:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.103+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d2010a2e0 con 0x7f3d200f5950 2023-11-16T01:16:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.103+0000 7f3d127fc700 1 -- 172.21.15.134:0/1680200751 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d14010230 con 0x7f3d200f5950 2023-11-16T01:16:23.105 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:23.105 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:23.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 msgr2=0x7f3d0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 0x7f3d0c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3d1c0096a0 tx=0x7f3d1c006b40).stop 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 msgr2=0x7f3d20109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3d14013510 tx=0x7f3d1400b040).stop 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 shutdown_connections 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 --2- 172.21.15.134:0/1680200751 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d200f5950 0x7f3d20109410 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 >> 172.21.15.134:0/1680200751 conn(0x7f3d200f0260 msgr2=0x7f3d200f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 shutdown_connections 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:23.105+0000 7f3d27f3d700 1 -- 172.21.15.134:0/1680200751 wait complete. 2023-11-16T01:16:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:24.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:22.004755+0000 mgr.smithi133.xdyttl (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:24.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:23.104843+0000 mon.smithi133 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.134:0/1680200751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:24.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:24.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:25.720 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 -- 172.21.15.134:0/1618451707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 msgr2=0x7fb4180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 --2- 172.21.15.134:0/1618451707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4180f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb408009fb0 tx=0x7fb408009b30).stop 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 -- 172.21.15.134:0/1618451707 shutdown_connections 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 --2- 172.21.15.134:0/1618451707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4180f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 -- 172.21.15.134:0/1618451707 >> 172.21.15.134:0/1618451707 conn(0x7fb4180f1c70 msgr2=0x7fb4180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 -- 172.21.15.134:0/1618451707 shutdown_connections 2023-11-16T01:16:25.750 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.717+0000 7fb42074b700 1 -- 172.21.15.134:0/1618451707 wait complete. 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb42074b700 1 Processor -- start 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb42074b700 1 -- start start 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb42074b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb42074b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418106bc0 con 0x7fb4180f6510 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb41e4e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb41e4e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40870/0 (socket says 172.21.15.134:40870) 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb41e4e7700 1 -- 172.21.15.134:0/2701578555 learned_addr learned my addr 172.21.15.134:0/2701578555 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.718+0000 7fb41e4e7700 1 -- 172.21.15.134:0/2701578555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408005040 con 0x7fb4180f6510 2023-11-16T01:16:25.751 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb41e4e7700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb408013510 tx=0x7fb40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408009320 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb418106dc0 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb408013f00 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408014d90 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.719+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb418107250 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4080095f0 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418045ad0 con 0x7fb4180f6510 2023-11-16T01:16:25.753 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb40effd700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:25.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4080410a0 con 0x7fb4180f6510 2023-11-16T01:16:25.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb41dce6700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:25.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.720+0000 7fb41dce6700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 0x7fb404032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb4140096a0 tx=0x7fb414006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:25.754 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.724+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb408010230 con 0x7fb4180f6510 2023-11-16T01:16:25.934 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.933+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4181080a0 con 0x7fb4180f6510 2023-11-16T01:16:25.935 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.933+0000 7fb40effd700 1 -- 172.21.15.134:0/2701578555 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb408010230 con 0x7fb4180f6510 2023-11-16T01:16:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:25.935 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:25.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 msgr2=0x7fb404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:25.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 0x7fb404032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb4140096a0 tx=0x7fb414006b40).stop 2023-11-16T01:16:25.936 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 msgr2=0x7fb4181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb408013510 tx=0x7fb40800b040).stop 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 shutdown_connections 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb40402ff60 0x7fb404032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 --2- 172.21.15.134:0/2701578555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4180f6510 0x7fb4181066d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 >> 172.21.15.134:0/2701578555 conn(0x7fb4180f1c70 msgr2=0x7fb4180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 shutdown_connections 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:25.935+0000 7fb42074b700 1 -- 172.21.15.134:0/2701578555 wait complete. 2023-11-16T01:16:25.937 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:26.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:24.005113+0000 mgr.smithi133.xdyttl (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:27.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:26 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:25.934904+0000 mon.smithi133 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.134:0/2701578555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:27.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:27.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:28.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:26.005427+0000 mgr.smithi133.xdyttl (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- 172.21.15.134:0/2929805280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 msgr2=0x7f69080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 --2- 172.21.15.134:0/2929805280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69080f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f68fc009fb0 tx=0x7f68fc009b30).stop 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- 172.21.15.134:0/2929805280 shutdown_connections 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 --2- 172.21.15.134:0/2929805280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69080f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- 172.21.15.134:0/2929805280 >> 172.21.15.134:0/2929805280 conn(0x7f69080f1c70 msgr2=0x7f69080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- 172.21.15.134:0/2929805280 shutdown_connections 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- 172.21.15.134:0/2929805280 wait complete. 2023-11-16T01:16:28.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 Processor -- start 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.526+0000 7f690eebb700 1 -- start start 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690eebb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690eebb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908106bc0 con 0x7f69080f6510 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690cc57700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690cc57700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40880/0 (socket says 172.21.15.134:40880) 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690cc57700 1 -- 172.21.15.134:0/1692174325 learned_addr learned my addr 172.21.15.134:0/1692174325 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690cc57700 1 -- 172.21.15.134:0/1692174325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc005040 con 0x7f69080f6510 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690cc57700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f68fc013510 tx=0x7f68fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68fc009320 con 0x7f69080f6510 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68fc013f00 con 0x7f69080f6510 2023-11-16T01:16:28.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6908106dc0 con 0x7f69080f6510 2023-11-16T01:16:28.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68fc014d90 con 0x7f69080f6510 2023-11-16T01:16:28.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.527+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6908107250 con 0x7f69080f6510 2023-11-16T01:16:28.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.528+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68fc0263a0 con 0x7f69080f6510 2023-11-16T01:16:28.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.528+0000 7f69057fa700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 0x7f68f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:28.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.528+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6908045ad0 con 0x7f69080f6510 2023-11-16T01:16:28.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.528+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68fc041380 con 0x7f69080f6510 2023-11-16T01:16:28.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.529+0000 7f6907fff700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 0x7f68f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:28.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.529+0000 7f6907fff700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 0x7f68f8032470 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f68f40096a0 tx=0x7f68f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:28.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.533+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68fc01d380 con 0x7f69080f6510 2023-11-16T01:16:28.737 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.735+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6908108100 con 0x7f69080f6510 2023-11-16T01:16:28.738 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.736+0000 7f69057fa700 1 -- 172.21.15.134:0/1692174325 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68fc012330 con 0x7f69080f6510 2023-11-16T01:16:28.738 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:28.738 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 msgr2=0x7f68f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 0x7f68f8032470 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f68f40096a0 tx=0x7f68f4006b40).stop 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 msgr2=0x7f69081066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f68fc013510 tx=0x7f68fc00b040).stop 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 shutdown_connections 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f68f802ffb0 0x7f68f8032470 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:28.740 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 --2- 172.21.15.134:0/1692174325 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69080f6510 0x7f69081066d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.737+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 >> 172.21.15.134:0/1692174325 conn(0x7f69080f1c70 msgr2=0x7f69080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.738+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 shutdown_connections 2023-11-16T01:16:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:28.738+0000 7f690eebb700 1 -- 172.21.15.134:0/1692174325 wait complete. 2023-11-16T01:16:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:29.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:28 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:28.737559+0000 mon.smithi133 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.134:0/1692174325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:30.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:30.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:30.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:28.005765+0000 mgr.smithi133.xdyttl (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:30.078169+0000 mon.smithi133 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:16:31.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:30.079062+0000 mon.smithi133 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:16:31.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 -- 172.21.15.134:0/614467322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 msgr2=0x7f85e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:31.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/614467322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e00f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f85c8009fb0 tx=0x7f85c8009b30).stop 2023-11-16T01:16:31.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 -- 172.21.15.134:0/614467322 shutdown_connections 2023-11-16T01:16:31.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/614467322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e00f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:31.324 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 -- 172.21.15.134:0/614467322 >> 172.21.15.134:0/614467322 conn(0x7f85e00f0b70 msgr2=0x7f85e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 -- 172.21.15.134:0/614467322 shutdown_connections 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.316+0000 7f85e6e8b700 1 -- 172.21.15.134:0/614467322 wait complete. 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e6e8b700 1 Processor -- start 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e6e8b700 1 -- start start 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e6e8b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e6e8b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e0105b20 con 0x7f85e00f5410 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e4c27700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e4c27700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59432/0 (socket says 172.21.15.134:59432) 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.317+0000 7f85e4c27700 1 -- 172.21.15.134:0/1087637519 learned_addr learned my addr 172.21.15.134:0/1087637519 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85e4c27700 1 -- 172.21.15.134:0/1087637519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8005040 con 0x7f85e00f5410 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85e4c27700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f85c8013510 tx=0x7f85c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:31.325 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85c8009320 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85c8013f00 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e0105d20 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85c8014d90 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.318+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e01061b0 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.319+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85c80263a0 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.319+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0045ad0 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.319+0000 7f85dd7fa700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.319+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85c8040fe0 con 0x7f85e00f5410 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.319+0000 7f85dffff700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.320+0000 7f85dffff700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 0x7f85cc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:31.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.324+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85c806d2b0 con 0x7f85e00f5410 2023-11-16T01:16:31.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.527+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85e00faf50 con 0x7f85e00f5410 2023-11-16T01:16:31.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.528+0000 7f85dd7fa700 1 -- 172.21.15.134:0/1087637519 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85c80126f0 con 0x7f85e00f5410 2023-11-16T01:16:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:31.530 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 msgr2=0x7f85cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 0x7f85cc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f85d00096a0 tx=0x7f85d0006b40).stop 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 msgr2=0x7f85e0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f85c8013510 tx=0x7f85c800b040).stop 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 shutdown_connections 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f85cc02ff60 0x7f85cc032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.529+0000 7f85e6e8b700 1 --2- 172.21.15.134:0/1087637519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85e00f5410 0x7f85e0105630 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.530+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 >> 172.21.15.134:0/1087637519 conn(0x7f85e00f0b70 msgr2=0x7f85e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.530+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 shutdown_connections 2023-11-16T01:16:31.532 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:31.530+0000 7f85e6e8b700 1 -- 172.21.15.134:0/1087637519 wait complete. 2023-11-16T01:16:31.533 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:32.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:30.006139+0000 mgr.smithi133.xdyttl (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:32.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:31.529417+0000 mon.smithi133 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.134:0/1087637519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:32.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:32.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:34.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.169+0000 7f852f0b9700 1 -- 172.21.15.134:0/1456817396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 msgr2=0x7f85280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.169+0000 7f852f0b9700 1 --2- 172.21.15.134:0/1456817396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85280f55c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f8524009fb0 tx=0x7f8524009b30).stop 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- 172.21.15.134:0/1456817396 shutdown_connections 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 --2- 172.21.15.134:0/1456817396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85280f55c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- 172.21.15.134:0/1456817396 >> 172.21.15.134:0/1456817396 conn(0x7f85280f09a0 msgr2=0x7f85280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- 172.21.15.134:0/1456817396 shutdown_connections 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- 172.21.15.134:0/1456817396 wait complete. 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 Processor -- start 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- start start 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852f0b9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85281069b0 con 0x7f85280f51a0 2023-11-16T01:16:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852e0b7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852e0b7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59454/0 (socket says 172.21.15.134:59454) 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.170+0000 7f852e0b7700 1 -- 172.21.15.134:0/2851549246 learned_addr learned my addr 172.21.15.134:0/2851549246 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f852e0b7700 1 -- 172.21.15.134:0/2851549246 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524005040 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f852e0b7700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f8524013510 tx=0x7f852400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524009320 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8528106bb0 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8524013f00 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8524014dc0 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.171+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8528107040 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.172+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85240095f0 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.172+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528045ad0 con 0x7f85280f51a0 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.172+0000 7f851effd700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.172+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8524041040 con 0x7f85280f51a0 2023-11-16T01:16:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.172+0000 7f852d8b6700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.173+0000 7f852d8b6700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f85180096a0 tx=0x7f8518006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.177+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8524014420 con 0x7f85280f51a0 2023-11-16T01:16:34.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:32.006518+0000 mgr.smithi133.xdyttl (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:34.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.385+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8528107ef0 con 0x7f85280f51a0 2023-11-16T01:16:34.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.385+0000 7f851effd700 1 -- 172.21.15.134:0/2851549246 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8524010230 con 0x7f85280f51a0 2023-11-16T01:16:34.387 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:34.387 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:34.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 msgr2=0x7f8514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:34.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f85180096a0 tx=0x7f8518006b40).stop 2023-11-16T01:16:34.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 msgr2=0x7f85281064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f8524013510 tx=0x7f852400b040).stop 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 shutdown_connections 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 --2- 172.21.15.134:0/2851549246 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85280f51a0 0x7f85281064c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 >> 172.21.15.134:0/2851549246 conn(0x7f85280f09a0 msgr2=0x7f85280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 shutdown_connections 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:34.387+0000 7f852f0b9700 1 -- 172.21.15.134:0/2851549246 wait complete. 2023-11-16T01:16:34.389 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:35.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:34 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:34.386959+0000 mon.smithi133 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.134:0/2851549246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:35.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:35.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:36.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:34.006861+0000 mgr.smithi133.xdyttl (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 -- 172.21.15.134:0/367109780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 msgr2=0x7f49280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 --2- 172.21.15.134:0/367109780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f49280f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f4910009fb0 tx=0x7f4910009b30).stop 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 -- 172.21.15.134:0/367109780 shutdown_connections 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 --2- 172.21.15.134:0/367109780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f49280f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 -- 172.21.15.134:0/367109780 >> 172.21.15.134:0/367109780 conn(0x7f49280f0b70 msgr2=0x7f49280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 -- 172.21.15.134:0/367109780 shutdown_connections 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.026+0000 7f492dc70700 1 -- 172.21.15.134:0/367109780 wait complete. 2023-11-16T01:16:37.030 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f492dc70700 1 Processor -- start 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f492dc70700 1 -- start start 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f492dc70700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f492dc70700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4928105b20 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f49277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f49277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59470/0 (socket says 172.21.15.134:59470) 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.027+0000 7f49277fe700 1 -- 172.21.15.134:0/2125830232 learned_addr learned my addr 172.21.15.134:0/2125830232 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f49277fe700 1 -- 172.21.15.134:0/2125830232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910005040 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f49277fe700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f4910013510 tx=0x7f491000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4910009320 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4928105d20 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4910013f00 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4910014d90 con 0x7f49280f5410 2023-11-16T01:16:37.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.028+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49281061b0 con 0x7f49280f5410 2023-11-16T01:16:37.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.029+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4910014420 con 0x7f49280f5410 2023-11-16T01:16:37.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.029+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928045ad0 con 0x7f49280f5410 2023-11-16T01:16:37.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.029+0000 7f491ffff700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 0x7f4914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:37.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.029+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4910041080 con 0x7f49280f5410 2023-11-16T01:16:37.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.029+0000 7f4926ffd700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 0x7f4914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:37.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.030+0000 7f4926ffd700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 0x7f4914032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f49180096a0 tx=0x7f4918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:37.035 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.034+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49100095f0 con 0x7f49280f5410 2023-11-16T01:16:37.238 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.236+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49280fae60 con 0x7f49280f5410 2023-11-16T01:16:37.238 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.237+0000 7f491ffff700 1 -- 172.21.15.134:0/2125830232 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f491001db70 con 0x7f49280f5410 2023-11-16T01:16:37.239 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:37.239 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:37.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 msgr2=0x7f4914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:37.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 0x7f4914032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f49180096a0 tx=0x7f4918006b40).stop 2023-11-16T01:16:37.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 msgr2=0x7f4928105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:37.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f4910013510 tx=0x7f491000b040).stop 2023-11-16T01:16:37.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 shutdown_connections 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f491402ff60 0x7f4914032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 --2- 172.21.15.134:0/2125830232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49280f5410 0x7f4928105630 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 >> 172.21.15.134:0/2125830232 conn(0x7f49280f0b70 msgr2=0x7f49280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 shutdown_connections 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:37.239+0000 7f492dc70700 1 -- 172.21.15.134:0/2125830232 wait complete. 2023-11-16T01:16:37.241 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:38.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:36.007122+0000 mgr.smithi133.xdyttl (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:38.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:37.238470+0000 mon.smithi133 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.134:0/2125830232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:38.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:38.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:39.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 -- 172.21.15.134:0/3050144615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 msgr2=0x7f54a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 --2- 172.21.15.134:0/3050144615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a40f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5498009fb0 tx=0x7f5498009b30).stop 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 -- 172.21.15.134:0/3050144615 shutdown_connections 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 --2- 172.21.15.134:0/3050144615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a40f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 -- 172.21.15.134:0/3050144615 >> 172.21.15.134:0/3050144615 conn(0x7f54a40f1c70 msgr2=0x7f54a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 -- 172.21.15.134:0/3050144615 shutdown_connections 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.806+0000 7f54aba4b700 1 -- 172.21.15.134:0/3050144615 wait complete. 2023-11-16T01:16:39.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54aba4b700 1 Processor -- start 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54aba4b700 1 -- start start 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54aba4b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54aba4b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a4108dd0 con 0x7f54a40f6510 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54a97e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54a97e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60852/0 (socket says 172.21.15.134:60852) 2023-11-16T01:16:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54a97e7700 1 -- 172.21.15.134:0/481982838 learned_addr learned my addr 172.21.15.134:0/481982838 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.807+0000 7f54a97e7700 1 -- 172.21.15.134:0/481982838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498005040 con 0x7f54a40f6510 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f54a97e7700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5498013510 tx=0x7f549800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5498009320 con 0x7f54a40f6510 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a4108fd0 con 0x7f54a40f6510 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5498013f00 con 0x7f54a40f6510 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5498014d90 con 0x7f54a40f6510 2023-11-16T01:16:39.812 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.808+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a4109460 con 0x7f54a40f6510 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.809+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54980095f0 con 0x7f54a40f6510 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.809+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a4045ad0 con 0x7f54a40f6510 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.809+0000 7f5495ffb700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 0x7f5490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.809+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54980410c0 con 0x7f54a40f6510 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.809+0000 7f54a8fe6700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 0x7f5490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:39.813 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.810+0000 7f54a8fe6700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 0x7f5490032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f54a00096a0 tx=0x7f54a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:39.816 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:39.814+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f549801d7f0 con 0x7f54a40f6510 2023-11-16T01:16:40.023 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.021+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54a406b280 con 0x7f54a40f6510 2023-11-16T01:16:40.024 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.023+0000 7f5495ffb700 1 -- 172.21.15.134:0/481982838 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5498010230 con 0x7f54a40f6510 2023-11-16T01:16:40.024 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:40.024 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:40.025 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 msgr2=0x7f5490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 0x7f5490032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f54a00096a0 tx=0x7f54a0006b40).stop 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 msgr2=0x7f54a41088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5498013510 tx=0x7f549800b040).stop 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 shutdown_connections 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f549002ff60 0x7f5490032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 --2- 172.21.15.134:0/481982838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54a40f6510 0x7f54a41088e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 >> 172.21.15.134:0/481982838 conn(0x7f54a40f1c70 msgr2=0x7f54a40f4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 shutdown_connections 2023-11-16T01:16:40.026 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:40.024+0000 7f54aba4b700 1 -- 172.21.15.134:0/481982838 wait complete. 2023-11-16T01:16:40.027 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:40.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:38.007478+0000 mgr.smithi133.xdyttl (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:41.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:40 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:40.023397+0000 mon.smithi133 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.134:0/481982838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:41.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:41.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:42.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:40.007822+0000 mgr.smithi133.xdyttl (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 -- 172.21.15.134:0/1206601050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 msgr2=0x7f36880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 --2- 172.21.15.134:0/1206601050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f36880f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f3678009fb0 tx=0x7f3678009b30).stop 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 -- 172.21.15.134:0/1206601050 shutdown_connections 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 --2- 172.21.15.134:0/1206601050 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f36880f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 -- 172.21.15.134:0/1206601050 >> 172.21.15.134:0/1206601050 conn(0x7f36880f0b70 msgr2=0x7f36880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 -- 172.21.15.134:0/1206601050 shutdown_connections 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.656+0000 7f36900af700 1 -- 172.21.15.134:0/1206601050 wait complete. 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f36900af700 1 Processor -- start 2023-11-16T01:16:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f36900af700 1 -- start start 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f36900af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f36900af700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688106c00 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f368de4b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f368de4b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60864/0 (socket says 172.21.15.134:60864) 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.657+0000 7f368de4b700 1 -- 172.21.15.134:0/1791982459 learned_addr learned my addr 172.21.15.134:0/1791982459 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f368de4b700 1 -- 172.21.15.134:0/1791982459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3678005040 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f368de4b700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f3678014000 tx=0x7f367800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36780095f0 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3688106e00 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3678007e70 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3678014ea0 con 0x7f36880f5410 2023-11-16T01:16:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.658+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3688107290 con 0x7f36880f5410 2023-11-16T01:16:42.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3678025070 con 0x7f36880f5410 2023-11-16T01:16:42.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688045ad0 con 0x7f36880f5410 2023-11-16T01:16:42.662 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f367e7fc700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 0x7f3674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:42.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36780409f0 con 0x7f36880f5410 2023-11-16T01:16:42.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f368d64a700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 0x7f3674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:42.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.659+0000 7f368d64a700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 0x7f3674032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f36840096a0 tx=0x7f3684006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:42.665 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.664+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36780079c0 con 0x7f36880f5410 2023-11-16T01:16:42.868 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.865+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3688108140 con 0x7f36880f5410 2023-11-16T01:16:42.868 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.866+0000 7f367e7fc700 1 -- 172.21.15.134:0/1791982459 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f367800f300 con 0x7f36880f5410 2023-11-16T01:16:42.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:42.869 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 msgr2=0x7f3674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 0x7f3674032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f36840096a0 tx=0x7f3684006b40).stop 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 msgr2=0x7f3688106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f3678014000 tx=0x7f367800b040).stop 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 shutdown_connections 2023-11-16T01:16:42.870 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f367402ff60 0x7f3674032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 --2- 172.21.15.134:0/1791982459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36880f5410 0x7f3688106710 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 >> 172.21.15.134:0/1791982459 conn(0x7f36880f0b70 msgr2=0x7f368805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 shutdown_connections 2023-11-16T01:16:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:42.868+0000 7f36900af700 1 -- 172.21.15.134:0/1791982459 wait complete. 2023-11-16T01:16:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:43.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:42 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:42.867749+0000 mon.smithi133 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.134:0/1791982459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:44.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:44.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:44.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:42.008101+0000 mgr.smithi133.xdyttl (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:45.503 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1481104922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 msgr2=0x7fd2ac0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:45.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1481104922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac0f5750 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fd2a0009fb0 tx=0x7fd2a0009b30).stop 2023-11-16T01:16:45.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1481104922 shutdown_connections 2023-11-16T01:16:45.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1481104922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac0f5750 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:45.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1481104922 >> 172.21.15.134:0/1481104922 conn(0x7fd2ac0f0a90 msgr2=0x7fd2ac0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:45.504 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1481104922 shutdown_connections 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.500+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1481104922 wait complete. 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2b0b7b700 1 Processor -- start 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2b0b7b700 1 -- start start 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2b0b7b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2b0b7b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac060240 con 0x7fd2ac0f5330 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60886/0 (socket says 172.21.15.134:60886) 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2aa59c700 1 -- 172.21.15.134:0/1080661666 learned_addr learned my addr 172.21.15.134:0/1080661666 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2aa59c700 1 -- 172.21.15.134:0/1080661666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a0005040 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.501+0000 7fd2aa59c700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fd2a0013510 tx=0x7fd2a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2a0009320 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2a0013f00 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2a0014d90 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac060440 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac109d80 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2a00095f0 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.502+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac045ad0 con 0x7fd2ac0f5330 2023-11-16T01:16:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.503+0000 7fd29affd700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 0x7fd294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:45.507 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.503+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2a00410c0 con 0x7fd2ac0f5330 2023-11-16T01:16:45.507 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.503+0000 7fd2a9d9b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 0x7fd294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:45.507 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.503+0000 7fd2a9d9b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 0x7fd294032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd29c0096a0 tx=0x7fd29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:45.509 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.507+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2a001d7f0 con 0x7fd2ac0f5330 2023-11-16T01:16:45.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.709+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2ac05ea50 con 0x7fd2ac0f5330 2023-11-16T01:16:45.710 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.709+0000 7fd29affd700 1 -- 172.21.15.134:0/1080661666 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd2a0010230 con 0x7fd2ac0f5330 2023-11-16T01:16:45.710 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:45.710 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:45.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 msgr2=0x7fd294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:45.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 0x7fd294032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd29c0096a0 tx=0x7fd29c006b40).stop 2023-11-16T01:16:45.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 msgr2=0x7fd2ac1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fd2a0013510 tx=0x7fd2a000b040).stop 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 shutdown_connections 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fd29402ff60 0x7fd294032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 --2- 172.21.15.134:0/1080661666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2ac0f5330 0x7fd2ac1096d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 >> 172.21.15.134:0/1080661666 conn(0x7fd2ac0f0a90 msgr2=0x7fd2ac0f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 shutdown_connections 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:45.711+0000 7fd2b0b7b700 1 -- 172.21.15.134:0/1080661666 wait complete. 2023-11-16T01:16:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:44.008419+0000 mgr.smithi133.xdyttl (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:46.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:45.710094+0000 mon.smithi133 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.134:0/1080661666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:47.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:47.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:48.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:46.008725+0000 mgr.smithi133.xdyttl (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:48.423 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- 172.21.15.134:0/1129010093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 msgr2=0x7f543c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:48.423 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 --2- 172.21.15.134:0/1129010093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f5438009fb0 tx=0x7f5438009b30).stop 2023-11-16T01:16:48.423 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- 172.21.15.134:0/1129010093 shutdown_connections 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 --2- 172.21.15.134:0/1129010093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- 172.21.15.134:0/1129010093 >> 172.21.15.134:0/1129010093 conn(0x7f543c0f0b70 msgr2=0x7f543c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- 172.21.15.134:0/1129010093 shutdown_connections 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- 172.21.15.134:0/1129010093 wait complete. 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 Processor -- start 2023-11-16T01:16:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- start start 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.419+0000 7f5444345700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c060240 con 0x7f543c0f5410 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f54420e1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f54420e1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:60898/0 (socket says 172.21.15.134:60898) 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f54420e1700 1 -- 172.21.15.134:0/2705486326 learned_addr learned my addr 172.21.15.134:0/2705486326 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f54420e1700 1 -- 172.21.15.134:0/2705486326 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438005040 con 0x7f543c0f5410 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f54420e1700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438009320 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c060440 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5438013f00 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438014d90 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.420+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c109ae0 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.421+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54380095f0 con 0x7f543c0f5410 2023-11-16T01:16:48.426 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.421+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c045ad0 con 0x7f543c0f5410 2023-11-16T01:16:48.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.421+0000 7f5432ffd700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 0x7f5428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:48.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.421+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54380410c0 con 0x7f543c0f5410 2023-11-16T01:16:48.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.421+0000 7f54418e0700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 0x7f5428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:48.427 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.422+0000 7f54418e0700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 0x7f5428032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f542c0096a0 tx=0x7f542c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:48.428 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.426+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54380263a0 con 0x7f543c0f5410 2023-11-16T01:16:48.634 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.630+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f543c0fae60 con 0x7f543c0f5410 2023-11-16T01:16:48.635 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.631+0000 7f5432ffd700 1 -- 172.21.15.134:0/2705486326 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5438010230 con 0x7f543c0f5410 2023-11-16T01:16:48.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:48.636 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 msgr2=0x7f5428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 0x7f5428032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f542c0096a0 tx=0x7f542c006b40).stop 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 msgr2=0x7f543c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).stop 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 shutdown_connections 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f542802ff60 0x7f5428032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 --2- 172.21.15.134:0/2705486326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f543c0f5410 0x7f543c109430 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.632+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 >> 172.21.15.134:0/2705486326 conn(0x7f543c0f0b70 msgr2=0x7f543c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.633+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 shutdown_connections 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:48.633+0000 7f5444345700 1 -- 172.21.15.134:0/2705486326 wait complete. 2023-11-16T01:16:48.641 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:49.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:48.631537+0000 mon.smithi133 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.134:0/2705486326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:50.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:50.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:50.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:49 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:48.009023+0000 mgr.smithi133.xdyttl (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:51.173 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 -- 172.21.15.134:0/3658084861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 msgr2=0x7fc08c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 --2- 172.21.15.134:0/3658084861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c0f90f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fc080009fb0 tx=0x7fc080009b30).stop 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 -- 172.21.15.134:0/3658084861 shutdown_connections 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 --2- 172.21.15.134:0/3658084861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c0f90f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 -- 172.21.15.134:0/3658084861 >> 172.21.15.134:0/3658084861 conn(0x7fc08c0f0b70 msgr2=0x7fc08c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 -- 172.21.15.134:0/3658084861 shutdown_connections 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 -- 172.21.15.134:0/3658084861 wait complete. 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.172+0000 7fc091e2d700 1 Processor -- start 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc091e2d700 1 -- start start 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc091e2d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc091e2d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c106b80 con 0x7fc08c060280 2023-11-16T01:16:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc08b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:51.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc08b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38324/0 (socket says 172.21.15.134:38324) 2023-11-16T01:16:51.175 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.173+0000 7fc08b7fe700 1 -- 172.21.15.134:0/615418720 learned_addr learned my addr 172.21.15.134:0/615418720 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc08b7fe700 1 -- 172.21.15.134:0/615418720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080005040 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc08b7fe700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc080013510 tx=0x7fc08000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc080009320 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c106d80 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc080013f00 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc080014dc0 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.174+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c107210 con 0x7fc08c060280 2023-11-16T01:16:51.176 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.175+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc080014420 con 0x7fc08c060280 2023-11-16T01:16:51.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.175+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c045ad0 con 0x7fc08c060280 2023-11-16T01:16:51.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.175+0000 7fc073fff700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:51.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.175+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc080041050 con 0x7fc08c060280 2023-11-16T01:16:51.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.175+0000 7fc08affd700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:51.177 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.176+0000 7fc08affd700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 0x7fc074032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc07c0096a0 tx=0x7fc07c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:51.180 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.180+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0800095f0 con 0x7fc08c060280 2023-11-16T01:16:51.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.384+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc08c1080c0 con 0x7fc08c060280 2023-11-16T01:16:51.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.385+0000 7fc073fff700 1 -- 172.21.15.134:0/615418720 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc08001cb10 con 0x7fc08c060280 2023-11-16T01:16:51.386 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:51.386 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 msgr2=0x7fc074032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 0x7fc074032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc07c0096a0 tx=0x7fc07c006b40).stop 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 msgr2=0x7fc08c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc080013510 tx=0x7fc08000b040).stop 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 shutdown_connections 2023-11-16T01:16:51.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fc07402ff60 0x7fc074032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:51.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 --2- 172.21.15.134:0/615418720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc08c060280 0x7fc08c106690 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:51.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.386+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 >> 172.21.15.134:0/615418720 conn(0x7fc08c0f0b70 msgr2=0x7fc08c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:51.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.387+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 shutdown_connections 2023-11-16T01:16:51.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:51.387+0000 7fc091e2d700 1 -- 172.21.15.134:0/615418720 wait complete. 2023-11-16T01:16:51.389 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:52.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:50.009385+0000 mgr.smithi133.xdyttl (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:52.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:51 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:51.385349+0000 mon.smithi133 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.134:0/615418720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:52.766 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:52.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:53.949 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 -- 172.21.15.134:0/1979576832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 msgr2=0x7f961c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:53.949 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 --2- 172.21.15.134:0/1979576832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c0f6e20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f9610009fb0 tx=0x7f9610009b30).stop 2023-11-16T01:16:53.949 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 -- 172.21.15.134:0/1979576832 shutdown_connections 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 --2- 172.21.15.134:0/1979576832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c0f6e20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 -- 172.21.15.134:0/1979576832 >> 172.21.15.134:0/1979576832 conn(0x7f961c0f0260 msgr2=0x7f961c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 -- 172.21.15.134:0/1979576832 shutdown_connections 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.948+0000 7f9622f29700 1 -- 172.21.15.134:0/1979576832 wait complete. 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9622f29700 1 Processor -- start 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9622f29700 1 -- start start 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9622f29700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:53.950 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9622f29700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c106b60 con 0x7f961c0f6a00 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9620cc5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9620cc5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38336/0 (socket says 172.21.15.134:38336) 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.949+0000 7f9620cc5700 1 -- 172.21.15.134:0/3621445986 learned_addr learned my addr 172.21.15.134:0/3621445986 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f9620cc5700 1 -- 172.21.15.134:0/3621445986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610005040 con 0x7f961c0f6a00 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f9620cc5700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9610013510 tx=0x7f961000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9610009320 con 0x7f961c0f6a00 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9610013f00 con 0x7f961c0f6a00 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c106d60 con 0x7f961c0f6a00 2023-11-16T01:16:53.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9610014dc0 con 0x7f961c0f6a00 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.950+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c1071f0 con 0x7f961c0f6a00 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f961001d3d0 con 0x7f961c0f6a00 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c045ad0 con 0x7f961c0f6a00 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f96197fa700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 0x7f960c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9610041320 con 0x7f961c0f6a00 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f961bfff700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 0x7f960c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:53.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.951+0000 7f961bfff700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 0x7f960c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:53.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:53.956+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96100263f0 con 0x7f961c0f6a00 2023-11-16T01:16:54.162 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.162+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f961c1080a0 con 0x7f961c0f6a00 2023-11-16T01:16:54.163 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.163+0000 7f96197fa700 1 -- 172.21.15.134:0/3621445986 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9610012330 con 0x7f961c0f6a00 2023-11-16T01:16:54.164 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:54.164 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 msgr2=0x7f960c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 0x7f960c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).stop 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 msgr2=0x7f961c106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9610013510 tx=0x7f961000b040).stop 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 shutdown_connections 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f960c02ff60 0x7f960c032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 --2- 172.21.15.134:0/3621445986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f961c0f6a00 0x7f961c106670 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 >> 172.21.15.134:0/3621445986 conn(0x7f961c0f0260 msgr2=0x7f961c0f4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:54.165 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 shutdown_connections 2023-11-16T01:16:54.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:54.164+0000 7f9622f29700 1 -- 172.21.15.134:0/3621445986 wait complete. 2023-11-16T01:16:54.166 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:54.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:53 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:52.009647+0000 mgr.smithi133.xdyttl (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:55.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:54 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:54.163031+0000 mon.smithi133 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.134:0/3621445986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:55.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:55.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:56.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:55 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:54.010005+0000 mgr.smithi133.xdyttl (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:56.705 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 -- 172.21.15.134:0/827195216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 msgr2=0x7f35180f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:56.705 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 --2- 172.21.15.134:0/827195216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35180f4be0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f3508009fb0 tx=0x7f3508009b30).stop 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 -- 172.21.15.134:0/827195216 shutdown_connections 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 --2- 172.21.15.134:0/827195216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35180f4be0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 -- 172.21.15.134:0/827195216 >> 172.21.15.134:0/827195216 conn(0x7f35180f02e0 msgr2=0x7f35180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 -- 172.21.15.134:0/827195216 shutdown_connections 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.704+0000 7f351d8e4700 1 -- 172.21.15.134:0/827195216 wait complete. 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f351d8e4700 1 Processor -- start 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f351d8e4700 1 -- start start 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f351d8e4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f351d8e4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518106bc0 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f3516ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38346/0 (socket says 172.21.15.134:38346) 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f3516ffd700 1 -- 172.21.15.134:0/3371852720 learned_addr learned my addr 172.21.15.134:0/3371852720 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.705+0000 7f3516ffd700 1 -- 172.21.15.134:0/3371852720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508005040 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f3516ffd700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3508013510 tx=0x7f350800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3508009320 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3518106dc0 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3508013f00 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3508014d90 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.706+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3518107250 con 0x7f35180f47c0 2023-11-16T01:16:56.708 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.707+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35080095f0 con 0x7f35180f47c0 2023-11-16T01:16:56.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.707+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3518045ad0 con 0x7f35180f47c0 2023-11-16T01:16:56.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.707+0000 7f34ff7fe700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 0x7f3500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:56.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.707+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3508040fe0 con 0x7f35180f47c0 2023-11-16T01:16:56.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.707+0000 7f35167fc700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 0x7f3500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:56.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.708+0000 7f35167fc700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 0x7f3500032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:56.712 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.712+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f350801d7f0 con 0x7f35180f47c0 2023-11-16T01:16:56.921 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.920+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3518108100 con 0x7f35180f47c0 2023-11-16T01:16:56.921 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.921+0000 7f34ff7fe700 1 -- 172.21.15.134:0/3371852720 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3508010230 con 0x7f35180f47c0 2023-11-16T01:16:56.922 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:56.922 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 msgr2=0x7f3500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 0x7f3500032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).stop 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 msgr2=0x7f35181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3508013510 tx=0x7f350800b040).stop 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 shutdown_connections 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f350002ff60 0x7f3500032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 --2- 172.21.15.134:0/3371852720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35180f47c0 0x7f35181066d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 >> 172.21.15.134:0/3371852720 conn(0x7f35180f02e0 msgr2=0x7f35180f0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.922+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 shutdown_connections 2023-11-16T01:16:56.925 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:56.923+0000 7f351d8e4700 1 -- 172.21.15.134:0/3371852720 wait complete. 2023-11-16T01:16:56.926 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:16:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:56.010323+0000 mgr.smithi133.xdyttl (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:16:58.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:57 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:56.921582+0000 mon.smithi133 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.134:0/3371852720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:16:58.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:16:58.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.565+0000 7f80f77e4700 1 -- 172.21.15.134:0/3041624281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f00f5410 msgr2=0x7f80f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.565+0000 7f80f77e4700 1 --2- 172.21.15.134:0/3041624281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f00f5410 0x7f80f00f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f80e0009fb0 tx=0x7f80e0009b30).stop 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- 172.21.15.134:0/3041624281 shutdown_connections 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 --2- 172.21.15.134:0/3041624281 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f00f5410 0x7f80f00f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- 172.21.15.134:0/3041624281 >> 172.21.15.134:0/3041624281 conn(0x7f80f00f0b70 msgr2=0x7f80f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- 172.21.15.134:0/3041624281 shutdown_connections 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- 172.21.15.134:0/3041624281 wait complete. 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 Processor -- start 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- start start 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:59.568 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f77e4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f0107150 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f5580700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f5580700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:40996/0 (socket says 172.21.15.134:40996) 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.566+0000 7f80f5580700 1 -- 172.21.15.134:0/2808069703 learned_addr learned my addr 172.21.15.134:0/2808069703 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80f5580700 1 -- 172.21.15.134:0/2808069703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0005040 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80f5580700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f80e0009f80 tx=0x7f80e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80e00136c0 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80e0013e70 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f0107350 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80e0014420 con 0x7f80f0106840 2023-11-16T01:16:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.567+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f0107ea0 con 0x7f80f0106840 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.568+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80e0025060 con 0x7f80f0106840 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.568+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f0059100 con 0x7f80f0106840 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.568+0000 7f80e5ffb700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 0x7f80dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.568+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80e0040e40 con 0x7f80f0106840 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.568+0000 7f80f4d7f700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 0x7f80dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:16:59.570 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.569+0000 7f80f4d7f700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 0x7f80dc032470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f80ec0096a0 tx=0x7f80ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:16:59.580 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.573+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80e001c950 con 0x7f80f0106840 2023-11-16T01:16:59.777 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.776+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80f0108660 con 0x7f80f0106840 2023-11-16T01:16:59.778 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.777+0000 7f80e5ffb700 1 -- 172.21.15.134:0/2808069703 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80e002d2d0 con 0x7f80f0106840 2023-11-16T01:16:59.778 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:16:59.778 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 msgr2=0x7f80dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 0x7f80dc032470 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f80ec0096a0 tx=0x7f80ec006b40).stop 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 msgr2=0x7f80f0106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f80e0009f80 tx=0x7f80e0009b30).stop 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 shutdown_connections 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f80dc02ffb0 0x7f80dc032470 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 --2- 172.21.15.134:0/2808069703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f80f0106840 0x7f80f0106c60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:16:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 >> 172.21.15.134:0/2808069703 conn(0x7f80f00f0b70 msgr2=0x7f80f005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:16:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 shutdown_connections 2023-11-16T01:16:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:16:59.779+0000 7f80f77e4700 1 -- 172.21.15.134:0/2808069703 wait complete. 2023-11-16T01:16:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:00.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:16:59 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:16:58.010670+0000 mgr.smithi133.xdyttl (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:01.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:00 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:16:59.777364+0000 mon.smithi133 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.134:0/2808069703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:01.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:01.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:02.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:01 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:00.010992+0000 mgr.smithi133.xdyttl (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:02.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 -- 172.21.15.134:0/1796701621 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 msgr2=0x7f90740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:02.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 --2- 172.21.15.134:0/1796701621 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f90740f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9064009fb0 tx=0x7f9064009b30).stop 2023-11-16T01:17:02.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 -- 172.21.15.134:0/1796701621 shutdown_connections 2023-11-16T01:17:02.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 --2- 172.21.15.134:0/1796701621 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f90740f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:02.449 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 -- 172.21.15.134:0/1796701621 >> 172.21.15.134:0/1796701621 conn(0x7f90740f0b70 msgr2=0x7f90740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 -- 172.21.15.134:0/1796701621 shutdown_connections 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 -- 172.21.15.134:0/1796701621 wait complete. 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.447+0000 7f907a2c2700 1 Processor -- start 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f907a2c2700 1 -- start start 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f907a2c2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:02.450 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f907a2c2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074060240 con 0x7f90740f5410 2023-11-16T01:17:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f9073fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f9073fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41022/0 (socket says 172.21.15.134:41022) 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f9073fff700 1 -- 172.21.15.134:0/4106815383 learned_addr learned my addr 172.21.15.134:0/4106815383 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.448+0000 7f9073fff700 1 -- 172.21.15.134:0/4106815383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064005040 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f9073fff700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9064013510 tx=0x7f906400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9064009320 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9074060440 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9064013f00 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9064014d90 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.449+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074109ae0 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.450+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90640095f0 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.450+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9074045ad0 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.450+0000 7f9070ff9700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 0x7f905c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.450+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9064041000 con 0x7f90740f5410 2023-11-16T01:17:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.450+0000 7f90737fe700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 0x7f905c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:02.453 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.451+0000 7f90737fe700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 0x7f905c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f90680096a0 tx=0x7f9068006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:02.456 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.455+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9064014420 con 0x7f90740f5410 2023-11-16T01:17:02.660 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.659+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90740fae60 con 0x7f90740f5410 2023-11-16T01:17:02.661 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.659+0000 7f9070ff9700 1 -- 172.21.15.134:0/4106815383 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9064010230 con 0x7f90740f5410 2023-11-16T01:17:02.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:02.661 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 msgr2=0x7f905c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 0x7f905c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f90680096a0 tx=0x7f9068006b40).stop 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 msgr2=0x7f9074109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9064013510 tx=0x7f906400b040).stop 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 shutdown_connections 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f905c02ff60 0x7f905c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 --2- 172.21.15.134:0/4106815383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90740f5410 0x7f9074109430 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 >> 172.21.15.134:0/4106815383 conn(0x7f90740f0b70 msgr2=0x7f90740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 shutdown_connections 2023-11-16T01:17:02.663 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:02.661+0000 7f907a2c2700 1 -- 172.21.15.134:0/4106815383 wait complete. 2023-11-16T01:17:02.664 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:03.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:02 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:02.660900+0000 mon.smithi133 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.134:0/4106815383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:04.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:03 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:02.011304+0000 mgr.smithi133.xdyttl (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:04.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:04.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:05.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- 172.21.15.134:0/1193777350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 msgr2=0x7f465c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:05.327 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 --2- 172.21.15.134:0/1193777350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c0f8f80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4658009fb0 tx=0x7f4658009b30).stop 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- 172.21.15.134:0/1193777350 shutdown_connections 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 --2- 172.21.15.134:0/1193777350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c0f8f80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- 172.21.15.134:0/1193777350 >> 172.21.15.134:0/1193777350 conn(0x7f465c0f09a0 msgr2=0x7f465c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- 172.21.15.134:0/1193777350 shutdown_connections 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- 172.21.15.134:0/1193777350 wait complete. 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 Processor -- start 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.325+0000 7f466307c700 1 -- start start 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466307c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466307c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c1069f0 con 0x7f465c060180 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466207a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466207a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41036/0 (socket says 172.21.15.134:41036) 2023-11-16T01:17:05.328 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466207a700 1 -- 172.21.15.134:0/3001873642 learned_addr learned my addr 172.21.15.134:0/3001873642 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:05.329 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466207a700 1 -- 172.21.15.134:0/3001873642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658005040 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466207a700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4658013510 tx=0x7f465800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4658009320 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c106bf0 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4658013f00 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4658014d90 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.326+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c107080 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.327+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46580095f0 con 0x7f465c060180 2023-11-16T01:17:05.330 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.327+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c045ad0 con 0x7f465c060180 2023-11-16T01:17:05.331 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.327+0000 7f4652ffd700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 0x7f4648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:05.331 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.327+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4658041000 con 0x7f465c060180 2023-11-16T01:17:05.331 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.328+0000 7f4661879700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 0x7f4648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:05.331 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.328+0000 7f4661879700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 0x7f4648032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f464c0096a0 tx=0x7f464c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:05.333 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.332+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f465801d7f0 con 0x7f465c060180 2023-11-16T01:17:05.537 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.535+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f465c107f30 con 0x7f465c060180 2023-11-16T01:17:05.538 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.536+0000 7f4652ffd700 1 -- 172.21.15.134:0/3001873642 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4658010230 con 0x7f465c060180 2023-11-16T01:17:05.538 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:05.538 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:05.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 msgr2=0x7f4648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:05.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 0x7f4648032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f464c0096a0 tx=0x7f464c006b40).stop 2023-11-16T01:17:05.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 msgr2=0x7f465c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:05.539 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4658013510 tx=0x7f465800b040).stop 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 shutdown_connections 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f464802ff60 0x7f4648032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 --2- 172.21.15.134:0/3001873642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c060180 0x7f465c106500 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 >> 172.21.15.134:0/3001873642 conn(0x7f465c0f09a0 msgr2=0x7f465c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 shutdown_connections 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:05.538+0000 7f466307c700 1 -- 172.21.15.134:0/3001873642 wait complete. 2023-11-16T01:17:05.540 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:04.011664+0000 mgr.smithi133.xdyttl (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:06.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:05 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:05.537613+0000 mon.smithi133 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.134:0/3001873642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:06.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:06.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:08.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:07 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:06.012027+0000 mgr.smithi133.xdyttl (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.143+0000 7f8f25470700 1 -- 172.21.15.134:0/1685320498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 msgr2=0x7f8f200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.143+0000 7f8f25470700 1 --2- 172.21.15.134:0/1685320498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f200f66c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f8f10009fb0 tx=0x7f8f10009b30).stop 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.143+0000 7f8f25470700 1 -- 172.21.15.134:0/1685320498 shutdown_connections 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.143+0000 7f8f25470700 1 --2- 172.21.15.134:0/1685320498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f200f66c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.143+0000 7f8f25470700 1 -- 172.21.15.134:0/1685320498 >> 172.21.15.134:0/1685320498 conn(0x7f8f200f1ac0 msgr2=0x7f8f200f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:08.146 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 -- 172.21.15.134:0/1685320498 shutdown_connections 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 -- 172.21.15.134:0/1685320498 wait complete. 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 Processor -- start 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 -- start start 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.144+0000 7f8f25470700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20106990 con 0x7f8f200f62a0 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41062/0 (socket says 172.21.15.134:41062) 2023-11-16T01:17:08.147 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1ffff700 1 -- 172.21.15.134:0/2871466171 learned_addr learned my addr 172.21.15.134:0/2871466171 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:08.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1ffff700 1 -- 172.21.15.134:0/2871466171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10005040 con 0x7f8f200f62a0 2023-11-16T01:17:08.148 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1ffff700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f8f10009830 tx=0x7f8f1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f10009320 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f100136c0 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f20106b90 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f1001ee90 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.145+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f20107020 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.146+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f1001e420 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.146+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f20045ad0 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.146+0000 7f8f1cff9700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.146+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f100410b0 con 0x7f8f200f62a0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.146+0000 7f8f1f7fe700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:08.149 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.147+0000 7f8f1f7fe700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 0x7f8f080323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:08.152 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.151+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f100095f0 con 0x7f8f200f62a0 2023-11-16T01:17:08.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.355+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f20107ed0 con 0x7f8f200f62a0 2023-11-16T01:17:08.357 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.356+0000 7f8f1cff9700 1 -- 172.21.15.134:0/2871466171 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f1002c2d0 con 0x7f8f200f62a0 2023-11-16T01:17:08.358 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:08.358 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 msgr2=0x7f8f080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 0x7f8f080323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).stop 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 msgr2=0x7f8f201064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f8f10009830 tx=0x7f8f1000b040).stop 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 shutdown_connections 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:08.359 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 --2- 172.21.15.134:0/2871466171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8f200f62a0 0x7f8f201064a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:08.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 >> 172.21.15.134:0/2871466171 conn(0x7f8f200f1ac0 msgr2=0x7f8f200f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:08.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 shutdown_connections 2023-11-16T01:17:08.360 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:08.357+0000 7f8f25470700 1 -- 172.21.15.134:0/2871466171 wait complete. 2023-11-16T01:17:08.360 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:09.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:08 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:08.357445+0000 mon.smithi133 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.134:0/2871466171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:09.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:09.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:10.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:09 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:08.012411+0000 mgr.smithi133.xdyttl (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:10.952 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- 172.21.15.134:0/2153547027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 msgr2=0x7fa43c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 --2- 172.21.15.134:0/2153547027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c0f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fa438009fb0 tx=0x7fa438009b30).stop 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- 172.21.15.134:0/2153547027 shutdown_connections 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 --2- 172.21.15.134:0/2153547027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c0f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- 172.21.15.134:0/2153547027 >> 172.21.15.134:0/2153547027 conn(0x7fa43c0f09a0 msgr2=0x7fa43c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- 172.21.15.134:0/2153547027 shutdown_connections 2023-11-16T01:17:10.953 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- 172.21.15.134:0/2153547027 wait complete. 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 Processor -- start 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.951+0000 7fa442289700 1 -- start start 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa442289700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa442289700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c1069e0 con 0x7fa43c0f51a0 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa441287700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa441287700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41674/0 (socket says 172.21.15.134:41674) 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa441287700 1 -- 172.21.15.134:0/3527108110 learned_addr learned my addr 172.21.15.134:0/3527108110 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa441287700 1 -- 172.21.15.134:0/3527108110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438005040 con 0x7fa43c0f51a0 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa441287700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa438013510 tx=0x7fa43800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa438009320 con 0x7fa43c0f51a0 2023-11-16T01:17:10.955 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c106be0 con 0x7fa43c0f51a0 2023-11-16T01:17:10.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa438013f00 con 0x7fa43c0f51a0 2023-11-16T01:17:10.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.952+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa438014dc0 con 0x7fa43c0f51a0 2023-11-16T01:17:10.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.953+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c107070 con 0x7fa43c0f51a0 2023-11-16T01:17:10.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.953+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4380095f0 con 0x7fa43c0f51a0 2023-11-16T01:17:10.956 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.953+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43c045ad0 con 0x7fa43c0f51a0 2023-11-16T01:17:10.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.954+0000 7fa431ffb700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 0x7fa428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:10.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.954+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa438041020 con 0x7fa43c0f51a0 2023-11-16T01:17:10.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.954+0000 7fa440a86700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 0x7fa428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:10.957 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.954+0000 7fa440a86700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 0x7fa428032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa42c0096a0 tx=0x7fa42c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:10.959 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:10.958+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa438014420 con 0x7fa43c0f51a0 2023-11-16T01:17:11.163 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.162+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa43c107ec0 con 0x7fa43c0f51a0 2023-11-16T01:17:11.164 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.163+0000 7fa431ffb700 1 -- 172.21.15.134:0/3527108110 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa43801c920 con 0x7fa43c0f51a0 2023-11-16T01:17:11.164 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:11.164 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 msgr2=0x7fa428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 0x7fa428032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa42c0096a0 tx=0x7fa42c006b40).stop 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 msgr2=0x7fa43c1064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa438013510 tx=0x7fa43800b040).stop 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 shutdown_connections 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa42802ff60 0x7fa428032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 --2- 172.21.15.134:0/3527108110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa43c0f51a0 0x7fa43c1064f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 >> 172.21.15.134:0/3527108110 conn(0x7fa43c0f09a0 msgr2=0x7fa43c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 shutdown_connections 2023-11-16T01:17:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:11.164+0000 7fa442289700 1 -- 172.21.15.134:0/3527108110 wait complete. 2023-11-16T01:17:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:12.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:10.012800+0000 mgr.smithi133.xdyttl (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:12.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:11 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:11.164142+0000 mon.smithi133 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.134:0/3527108110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:12.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:12.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:13.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 -- 172.21.15.134:0/673371861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e80f5410 msgr2=0x7f30e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:13.765 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/673371861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e80f5410 0x7f30e80f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f30d8009fb0 tx=0x7f30d8009b30).stop 2023-11-16T01:17:13.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 -- 172.21.15.134:0/673371861 shutdown_connections 2023-11-16T01:17:13.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/673371861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e80f5410 0x7f30e80f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:13.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 -- 172.21.15.134:0/673371861 >> 172.21.15.134:0/673371861 conn(0x7f30e80f0b70 msgr2=0x7f30e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:13.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 -- 172.21.15.134:0/673371861 shutdown_connections 2023-11-16T01:17:13.766 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.763+0000 7f30ee7d8700 1 -- 172.21.15.134:0/673371861 wait complete. 2023-11-16T01:17:13.767 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30ee7d8700 1 Processor -- start 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30ee7d8700 1 -- start start 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30ee7d8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30ee7d8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e8107130 con 0x7f30e8106820 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30e7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30e7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41702/0 (socket says 172.21.15.134:41702) 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.764+0000 7f30e7fff700 1 -- 172.21.15.134:0/2598342523 learned_addr learned my addr 172.21.15.134:0/2598342523 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30e7fff700 1 -- 172.21.15.134:0/2598342523 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d8005040 con 0x7f30e8106820 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30e7fff700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f30d8009f80 tx=0x7f30d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d80136c0 con 0x7f30e8106820 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e8107330 con 0x7f30e8106820 2023-11-16T01:17:13.768 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30d80095f0 con 0x7f30e8106820 2023-11-16T01:17:13.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d8014420 con 0x7f30e8106820 2023-11-16T01:17:13.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.765+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e8107e80 con 0x7f30e8106820 2023-11-16T01:17:13.769 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30d8025060 con 0x7f30e8106820 2023-11-16T01:17:13.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30e57fa700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 0x7f30d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:13.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e8059100 con 0x7f30e8106820 2023-11-16T01:17:13.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30d8040da0 con 0x7f30e8106820 2023-11-16T01:17:13.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30dffff700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 0x7f30d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:13.770 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.766+0000 7f30dffff700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 0x7f30d4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f30d00096a0 tx=0x7f30d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:13.772 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.771+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30d801c950 con 0x7f30e8106820 2023-11-16T01:17:13.976 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.974+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30e8045ad0 con 0x7f30e8106820 2023-11-16T01:17:13.978 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.975+0000 7f30e57fa700 1 -- 172.21.15.134:0/2598342523 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30d801cb60 con 0x7f30e8106820 2023-11-16T01:17:13.979 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:13.979 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:13.979 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 msgr2=0x7f30d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:13.979 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 0x7f30d4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f30d00096a0 tx=0x7f30d0006b40).stop 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 msgr2=0x7f30e8106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f30d8009f80 tx=0x7f30d8009b30).stop 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 shutdown_connections 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f30d402ff60 0x7f30d4032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 --2- 172.21.15.134:0/2598342523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e8106820 0x7f30e8106c40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 >> 172.21.15.134:0/2598342523 conn(0x7f30e80f0b70 msgr2=0x7f30e80f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 shutdown_connections 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:13.977+0000 7f30ee7d8700 1 -- 172.21.15.134:0/2598342523 wait complete. 2023-11-16T01:17:13.980 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:14.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:13 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:12.013059+0000 mgr.smithi133.xdyttl (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:15.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:14 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:13.976532+0000 mon.smithi133 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.134:0/2598342523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:15.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:15.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:16.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:15 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:14.013410+0000 mgr.smithi133.xdyttl (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1628469491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 msgr2=0x7f1c580f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1628469491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c580f8f80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f1c48009fb0 tx=0x7f1c48009b30).stop 2023-11-16T01:17:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1628469491 shutdown_connections 2023-11-16T01:17:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1628469491 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c580f8f80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1628469491 >> 172.21.15.134:0/1628469491 conn(0x7f1c580f09a0 msgr2=0x7f1c580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1628469491 shutdown_connections 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1628469491 wait complete. 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.492+0000 7f1c5d5da700 1 Processor -- start 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c5d5da700 1 -- start start 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c5d5da700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c5d5da700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58106a20 con 0x7f1c58060180 2023-11-16T01:17:16.496 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c57fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c57fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:41726/0 (socket says 172.21.15.134:41726) 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.493+0000 7f1c57fff700 1 -- 172.21.15.134:0/1452195973 learned_addr learned my addr 172.21.15.134:0/1452195973 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c57fff700 1 -- 172.21.15.134:0/1452195973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48005040 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c57fff700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f1c48013510 tx=0x7f1c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c48009320 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c48013f00 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c58106c20 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c48014d90 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.494+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c581070b0 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c48014420 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58045ad0 con 0x7f1c58060180 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c54ff9700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:16.497 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c48041000 con 0x7f1c58060180 2023-11-16T01:17:16.498 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c577fe700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:16.498 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.495+0000 7f1c577fe700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 0x7f1c40032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1c4c0096a0 tx=0x7f1c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:16.501 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.500+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c480095f0 con 0x7f1c58060180 2023-11-16T01:17:16.706 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.705+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c58107f60 con 0x7f1c58060180 2023-11-16T01:17:16.707 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.706+0000 7f1c54ff9700 1 -- 172.21.15.134:0/1452195973 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c4801db70 con 0x7f1c58060180 2023-11-16T01:17:16.707 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:16.708 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 msgr2=0x7f1c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 0x7f1c40032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1c4c0096a0 tx=0x7f1c4c006b40).stop 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 msgr2=0x7f1c58106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f1c48013510 tx=0x7f1c4800b040).stop 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 shutdown_connections 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f1c4002ff60 0x7f1c40032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 --2- 172.21.15.134:0/1452195973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c58060180 0x7f1c58106530 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 >> 172.21.15.134:0/1452195973 conn(0x7f1c580f09a0 msgr2=0x7f1c580f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:16.709 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 shutdown_connections 2023-11-16T01:17:16.710 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:16.707+0000 7f1c5d5da700 1 -- 172.21.15.134:0/1452195973 wait complete. 2023-11-16T01:17:16.710 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:18.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:16.013665+0000 mgr.smithi133.xdyttl (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:18.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:17 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:16.707227+0000 mon.smithi133 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.134:0/1452195973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:18.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:18.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:19.278 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 -- 172.21.15.134:0/603039716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 msgr2=0x7f7e380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 --2- 172.21.15.134:0/603039716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e380f90f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f7e28009fb0 tx=0x7f7e28009b30).stop 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 -- 172.21.15.134:0/603039716 shutdown_connections 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 --2- 172.21.15.134:0/603039716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e380f90f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 -- 172.21.15.134:0/603039716 >> 172.21.15.134:0/603039716 conn(0x7f7e380f0b70 msgr2=0x7f7e380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 -- 172.21.15.134:0/603039716 shutdown_connections 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.276+0000 7f7e3db47700 1 -- 172.21.15.134:0/603039716 wait complete. 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e3db47700 1 Processor -- start 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e3db47700 1 -- start start 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e3db47700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e3db47700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e38106bd0 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e377fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e377fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38852/0 (socket says 172.21.15.134:38852) 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.277+0000 7f7e377fe700 1 -- 172.21.15.134:0/2883400713 learned_addr learned my addr 172.21.15.134:0/2883400713 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e377fe700 1 -- 172.21.15.134:0/2883400713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e28005040 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e377fe700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7e28013510 tx=0x7f7e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e28009320 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e38106dd0 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e28013f00 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e28014dc0 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.278+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e38107260 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.279+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e280095f0 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.279+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e38045ad0 con 0x7f7e38060280 2023-11-16T01:17:19.281 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.279+0000 7f7e2ffff700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 0x7f7e24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:19.282 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.279+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e28041020 con 0x7f7e38060280 2023-11-16T01:17:19.282 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.279+0000 7f7e36ffd700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 0x7f7e24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:19.282 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.280+0000 7f7e36ffd700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 0x7f7e24032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7e200096a0 tx=0x7f7e20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:19.286 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.284+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e28014420 con 0x7f7e38060280 2023-11-16T01:17:19.489 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.487+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e381080b0 con 0x7f7e38060280 2023-11-16T01:17:19.490 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.488+0000 7f7e2ffff700 1 -- 172.21.15.134:0/2883400713 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e2801c920 con 0x7f7e38060280 2023-11-16T01:17:19.490 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:19.490 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:19.491 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 msgr2=0x7f7e24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 0x7f7e24032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7e200096a0 tx=0x7f7e20006b40).stop 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 msgr2=0x7f7e381066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7e28013510 tx=0x7f7e2800b040).stop 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 shutdown_connections 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7e2402ff60 0x7f7e24032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 --2- 172.21.15.134:0/2883400713 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e38060280 0x7f7e381066e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 >> 172.21.15.134:0/2883400713 conn(0x7f7e380f0b70 msgr2=0x7f7e380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:19.492 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.489+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 shutdown_connections 2023-11-16T01:17:19.493 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:19.490+0000 7f7e3db47700 1 -- 172.21.15.134:0/2883400713 wait complete. 2023-11-16T01:17:19.493 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:18.013987+0000 mgr.smithi133.xdyttl (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:20.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:19 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:19.489511+0000 mon.smithi133 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.134:0/2883400713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:20.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:20.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:22.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.024+0000 7f6276d3a700 1 -- 172.21.15.134:0/2086161465 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 msgr2=0x7f62700f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:22.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.024+0000 7f6276d3a700 1 --2- 172.21.15.134:0/2086161465 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f62700f8f80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f6260009fb0 tx=0x7f6260009b30).stop 2023-11-16T01:17:22.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.024+0000 7f6276d3a700 1 -- 172.21.15.134:0/2086161465 shutdown_connections 2023-11-16T01:17:22.027 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.024+0000 7f6276d3a700 1 --2- 172.21.15.134:0/2086161465 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f62700f8f80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.024+0000 7f6276d3a700 1 -- 172.21.15.134:0/2086161465 >> 172.21.15.134:0/2086161465 conn(0x7f62700f09a0 msgr2=0x7f62700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 -- 172.21.15.134:0/2086161465 shutdown_connections 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 -- 172.21.15.134:0/2086161465 wait complete. 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 Processor -- start 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 -- start start 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:22.028 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6276d3a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62701069f0 con 0x7f6270060180 2023-11-16T01:17:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6275d38700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6275d38700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38872/0 (socket says 172.21.15.134:38872) 2023-11-16T01:17:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.025+0000 7f6275d38700 1 -- 172.21.15.134:0/1337775543 learned_addr learned my addr 172.21.15.134:0/1337775543 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f6275d38700 1 -- 172.21.15.134:0/1337775543 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260005040 con 0x7f6270060180 2023-11-16T01:17:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f6275d38700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f6260013510 tx=0x7f626000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6260009320 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6270106bf0 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6260013f00 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6260014d90 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.026+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6270107080 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.027+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62600263a0 con 0x7f6270060180 2023-11-16T01:17:22.031 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.027+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6270045ad0 con 0x7f6270060180 2023-11-16T01:17:22.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.027+0000 7f62667fc700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 0x7f625c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:22.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.027+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6260041310 con 0x7f6270060180 2023-11-16T01:17:22.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.027+0000 7f6275537700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 0x7f625c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:22.032 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.028+0000 7f6275537700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 0x7f625c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f626c0096a0 tx=0x7f626c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:22.032 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:21 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:20.014367+0000 mgr.smithi133.xdyttl (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:22.033 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.032+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f626001cb10 con 0x7f6270060180 2023-11-16T01:17:22.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.238+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6270107f30 con 0x7f6270060180 2023-11-16T01:17:22.240 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.239+0000 7f62667fc700 1 -- 172.21.15.134:0/1337775543 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f626001cb10 con 0x7f6270060180 2023-11-16T01:17:22.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:22.241 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 msgr2=0x7f625c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 0x7f625c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f626c0096a0 tx=0x7f626c006b40).stop 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 msgr2=0x7f6270106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f6260013510 tx=0x7f626000b040).stop 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 shutdown_connections 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f625c02ff60 0x7f625c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:22.242 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 --2- 172.21.15.134:0/1337775543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6270060180 0x7f6270106500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 >> 172.21.15.134:0/1337775543 conn(0x7f62700f09a0 msgr2=0x7f62700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 shutdown_connections 2023-11-16T01:17:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:22.240+0000 7f6276d3a700 1 -- 172.21.15.134:0/1337775543 wait complete. 2023-11-16T01:17:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:23.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:22 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:22.240231+0000 mon.smithi133 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.134:0/1337775543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:23.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:23.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:24.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:23 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:22.014752+0000 mgr.smithi133.xdyttl (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:24.805 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 -- 172.21.15.134:0/191493005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 msgr2=0x7ff8940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:24.805 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 --2- 172.21.15.134:0/191493005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff8940f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ff888009fb0 tx=0x7ff888009b30).stop 2023-11-16T01:17:24.805 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 -- 172.21.15.134:0/191493005 shutdown_connections 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 --2- 172.21.15.134:0/191493005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff8940f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 -- 172.21.15.134:0/191493005 >> 172.21.15.134:0/191493005 conn(0x7ff8940f1c70 msgr2=0x7ff8940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 -- 172.21.15.134:0/191493005 shutdown_connections 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 -- 172.21.15.134:0/191493005 wait complete. 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.803+0000 7ff89afb9700 1 Processor -- start 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff89afb9700 1 -- start start 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff89afb9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff89afb9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff894106c20 con 0x7ff8940f6510 2023-11-16T01:17:24.807 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff898d55700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff898d55700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38896/0 (socket says 172.21.15.134:38896) 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff898d55700 1 -- 172.21.15.134:0/493726829 learned_addr learned my addr 172.21.15.134:0/493726829 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff898d55700 1 -- 172.21.15.134:0/493726829 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff888005040 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.804+0000 7ff898d55700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff888013510 tx=0x7ff88800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff888009320 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff888013f00 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff894106e20 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff888014d90 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8941072b0 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8880263a0 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.805+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894045ad0 con 0x7ff8940f6510 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.806+0000 7ff8917fa700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 0x7ff884032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.806+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8880412b0 con 0x7ff8940f6510 2023-11-16T01:17:24.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.806+0000 7ff893fff700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 0x7ff884032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:24.809 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.806+0000 7ff893fff700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 0x7ff884032470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff8800096a0 tx=0x7ff880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:24.811 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:24.811+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff88801d380 con 0x7ff8940f6510 2023-11-16T01:17:25.018 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.016+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8940fbf60 con 0x7ff8940f6510 2023-11-16T01:17:25.018 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.018+0000 7ff8917fa700 1 -- 172.21.15.134:0/493726829 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff88801cb10 con 0x7ff8940f6510 2023-11-16T01:17:25.019 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:25.019 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.018+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 msgr2=0x7ff884032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.018+0000 7ff89afb9700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 0x7ff884032470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff8800096a0 tx=0x7ff880006b40).stop 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.018+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 msgr2=0x7ff894106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.018+0000 7ff89afb9700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff888013510 tx=0x7ff88800b040).stop 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 shutdown_connections 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7ff88402ffb0 0x7ff884032470 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 --2- 172.21.15.134:0/493726829 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8940f6510 0x7ff894106730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:25.020 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 >> 172.21.15.134:0/493726829 conn(0x7ff8940f1c70 msgr2=0x7ff8940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:25.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 shutdown_connections 2023-11-16T01:17:25.021 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:25.019+0000 7ff89afb9700 1 -- 172.21.15.134:0/493726829 wait complete. 2023-11-16T01:17:25.021 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:26.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:24.015098+0000 mgr.smithi133.xdyttl (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:26.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:25 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:25.018607+0000 mon.smithi133 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.134:0/493726829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:26.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:26.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- 172.21.15.134:0/2835859315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 msgr2=0x7f9ec80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/2835859315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec80f66c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9ec4009fb0 tx=0x7f9ec4009b30).stop 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- 172.21.15.134:0/2835859315 shutdown_connections 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/2835859315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec80f66c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- 172.21.15.134:0/2835859315 >> 172.21.15.134:0/2835859315 conn(0x7f9ec80f1aa0 msgr2=0x7f9ec80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- 172.21.15.134:0/2835859315 shutdown_connections 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- 172.21.15.134:0/2835859315 wait complete. 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 Processor -- start 2023-11-16T01:17:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- start start 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.635+0000 7f9ecf48d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec81069a0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ece48b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ece48b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:38912/0 (socket says 172.21.15.134:38912) 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ece48b700 1 -- 172.21.15.134:0/1882761821 learned_addr learned my addr 172.21.15.134:0/1882761821 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ece48b700 1 -- 172.21.15.134:0/1882761821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4005040 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ece48b700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9ec4013510 tx=0x7f9ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4009320 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec8106ba0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ec4013f00 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4014dc0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.638 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.636+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec8107030 con 0x7f9ec80f62a0 2023-11-16T01:17:27.639 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.637+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ec40095f0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.639 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.637+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec8045ad0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.640 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.637+0000 7f9ebeffd700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 0x7f9eb4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:27.640 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.637+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ec4040ff0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.640 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.637+0000 7f9ecdc8a700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 0x7f9eb4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:27.640 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.638+0000 7f9ecdc8a700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 0x7f9eb4032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9eb80096a0 tx=0x7f9eb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:27.642 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.642+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ec406d2b0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.849 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.849+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec8107ee0 con 0x7f9ec80f62a0 2023-11-16T01:17:27.850 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.849+0000 7f9ebeffd700 1 -- 172.21.15.134:0/1882761821 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ec4014420 con 0x7f9ec80f62a0 2023-11-16T01:17:27.850 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:27.850 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 msgr2=0x7f9eb4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 0x7f9eb4032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9eb80096a0 tx=0x7f9eb8006b40).stop 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 msgr2=0x7f9ec81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9ec4013510 tx=0x7f9ec400b040).stop 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 shutdown_connections 2023-11-16T01:17:27.852 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9eb402ffb0 0x7f9eb4032470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:27.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 --2- 172.21.15.134:0/1882761821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec80f62a0 0x7f9ec81064b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:27.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 >> 172.21.15.134:0/1882761821 conn(0x7f9ec80f1aa0 msgr2=0x7f9ec80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:27.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 shutdown_connections 2023-11-16T01:17:27.853 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:27.851+0000 7f9ecf48d700 1 -- 172.21.15.134:0/1882761821 wait complete. 2023-11-16T01:17:27.853 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:28.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:27 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:26.015455+0000 mgr.smithi133.xdyttl (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:29.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:28 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:27.849895+0000 mon.smithi133 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.134:0/1882761821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:29.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:29.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:30.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:29 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:28.015759+0000 mgr.smithi133.xdyttl (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 -- 172.21.15.134:0/1386397518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 msgr2=0x7fb5bc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/1386397518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc0f90f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb5b0009fb0 tx=0x7fb5b0009b30).stop 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 -- 172.21.15.134:0/1386397518 shutdown_connections 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/1386397518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc0f90f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 -- 172.21.15.134:0/1386397518 >> 172.21.15.134:0/1386397518 conn(0x7fb5bc0f0b70 msgr2=0x7fb5bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 -- 172.21.15.134:0/1386397518 shutdown_connections 2023-11-16T01:17:30.399 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.398+0000 7fb5c0b91700 1 -- 172.21.15.134:0/1386397518 wait complete. 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5c0b91700 1 Processor -- start 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5c0b91700 1 -- start start 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5c0b91700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5c0b91700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc106bd0 con 0x7fb5bc060280 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5ba59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5ba59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59760/0 (socket says 172.21.15.134:59760) 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5ba59c700 1 -- 172.21.15.134:0/25701978 learned_addr learned my addr 172.21.15.134:0/25701978 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5ba59c700 1 -- 172.21.15.134:0/25701978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0005040 con 0x7fb5bc060280 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.399+0000 7fb5ba59c700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb5b0013510 tx=0x7fb5b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b0009320 con 0x7fb5bc060280 2023-11-16T01:17:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5b0013f00 con 0x7fb5bc060280 2023-11-16T01:17:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc106dd0 con 0x7fb5bc060280 2023-11-16T01:17:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5b0014dc0 con 0x7fb5bc060280 2023-11-16T01:17:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc107260 con 0x7fb5bc060280 2023-11-16T01:17:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.400+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5b00095f0 con 0x7fb5bc060280 2023-11-16T01:17:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.401+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc045ad0 con 0x7fb5bc060280 2023-11-16T01:17:30.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.401+0000 7fb5aaffd700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 0x7fb5a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:30.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.401+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5b0041100 con 0x7fb5bc060280 2023-11-16T01:17:30.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.401+0000 7fb5b9d9b700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 0x7fb5a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:30.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.401+0000 7fb5b9d9b700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 0x7fb5a4032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb5ac0096a0 tx=0x7fb5ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:30.406 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.405+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5b0010230 con 0x7fb5bc060280 2023-11-16T01:17:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.612+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5bc108110 con 0x7fb5bc060280 2023-11-16T01:17:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.612+0000 7fb5aaffd700 1 -- 172.21.15.134:0/25701978 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb5b0010230 con 0x7fb5bc060280 2023-11-16T01:17:30.613 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:30.613 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 msgr2=0x7fb5a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 0x7fb5a4032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb5ac0096a0 tx=0x7fb5ac006b40).stop 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 msgr2=0x7fb5bc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb5b0013510 tx=0x7fb5b000b040).stop 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 shutdown_connections 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fb5a402ff60 0x7fb5a4032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 --2- 172.21.15.134:0/25701978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5bc060280 0x7fb5bc1066e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 >> 172.21.15.134:0/25701978 conn(0x7fb5bc0f0b70 msgr2=0x7fb5bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:30.615 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 shutdown_connections 2023-11-16T01:17:30.616 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:30.614+0000 7fb5c0b91700 1 -- 172.21.15.134:0/25701978 wait complete. 2023-11-16T01:17:30.616 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:30.079377+0000 mon.smithi133 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/mirror_snapshot_schedule"}]: dispatch 2023-11-16T01:17:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:30.080014+0000 mon.smithi133 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.133:0/4243765372' entity='mgr.smithi133.xdyttl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.xdyttl/trash_purge_schedule"}]: dispatch 2023-11-16T01:17:31.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:30 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:30.613029+0000 mon.smithi133 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.134:0/25701978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:31.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:31.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:32.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:31 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:30.016074+0000 mgr.smithi133.xdyttl (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:33.191 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 -- 172.21.15.134:0/4034663266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 msgr2=0x7f7d000f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:33.191 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 --2- 172.21.15.134:0/4034663266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f4be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f7cf0009fb0 tx=0x7f7cf0009b30).stop 2023-11-16T01:17:33.191 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 -- 172.21.15.134:0/4034663266 shutdown_connections 2023-11-16T01:17:33.191 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 --2- 172.21.15.134:0/4034663266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f4be0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:33.191 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 -- 172.21.15.134:0/4034663266 >> 172.21.15.134:0/4034663266 conn(0x7f7d000f02e0 msgr2=0x7f7d000f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:33.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 -- 172.21.15.134:0/4034663266 shutdown_connections 2023-11-16T01:17:33.192 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.190+0000 7f7d0546f700 1 -- 172.21.15.134:0/4034663266 wait complete. 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7d0546f700 1 Processor -- start 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7d0546f700 1 -- start start 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7d0546f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7d0546f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000f57f0 con 0x7f7d000f47c0 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7cfeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7cfeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59774/0 (socket says 172.21.15.134:59774) 2023-11-16T01:17:33.193 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.191+0000 7f7cfeffd700 1 -- 172.21.15.134:0/2119848088 learned_addr learned my addr 172.21.15.134:0/2119848088 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7cfeffd700 1 -- 172.21.15.134:0/2119848088 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0005040 con 0x7f7d000f47c0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7cfeffd700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f7cf0013510 tx=0x7f7cf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf0009320 con 0x7f7d000f47c0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d000f5990 con 0x7f7d000f47c0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf0013f00 con 0x7f7d000f47c0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf0014d90 con 0x7f7d000f47c0 2023-11-16T01:17:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.192+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d000f5e20 con 0x7f7d000f47c0 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.193+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cf00263a0 con 0x7f7d000f47c0 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.193+0000 7f7ce77fe700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.193+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cf00410f0 con 0x7f7d000f47c0 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.193+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00059100 con 0x7f7d000f47c0 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.193+0000 7f7cfe7fc700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.194+0000 7f7cfe7fc700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7cf40096a0 tx=0x7f7cf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:33.198 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.198+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cf0012920 con 0x7f7d000f47c0 2023-11-16T01:17:33.401 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.401+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d00045ad0 con 0x7f7d000f47c0 2023-11-16T01:17:33.403 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.402+0000 7f7ce77fe700 1 -- 172.21.15.134:0/2119848088 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7cf002b020 con 0x7f7d000f47c0 2023-11-16T01:17:33.403 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:33.403 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:33.404 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 msgr2=0x7f7ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7cf40096a0 tx=0x7f7cf4006b40).stop 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 msgr2=0x7f7d000f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f7cf0013510 tx=0x7f7cf000b040).stop 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 shutdown_connections 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 --2- 172.21.15.134:0/2119848088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d000f47c0 0x7f7d000f7370 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 >> 172.21.15.134:0/2119848088 conn(0x7f7d000f02e0 msgr2=0x7f7d000f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 shutdown_connections 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:33.403+0000 7f7d0546f700 1 -- 172.21.15.134:0/2119848088 wait complete. 2023-11-16T01:17:33.405 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:34.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:32.016416+0000 mgr.smithi133.xdyttl (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:34.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:33 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:33.402279+0000 mon.smithi133 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.134:0/2119848088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:34.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:34.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:36.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:35 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:34.016731+0000 mgr.smithi133.xdyttl (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.100+0000 7f9adc6b7700 1 -- 172.21.15.134:0/2298408911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 msgr2=0x7f9ad40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.100+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/2298408911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad40f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f9ac4009fb0 tx=0x7f9ac4009b30).stop 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- 172.21.15.134:0/2298408911 shutdown_connections 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/2298408911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad40f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- 172.21.15.134:0/2298408911 >> 172.21.15.134:0/2298408911 conn(0x7f9ad40f1c70 msgr2=0x7f9ad40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- 172.21.15.134:0/2298408911 shutdown_connections 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- 172.21.15.134:0/2298408911 wait complete. 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 Processor -- start 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- start start 2023-11-16T01:17:36.103 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9adc6b7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad4106c20 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9ada453700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9ada453700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59784/0 (socket says 172.21.15.134:59784) 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.101+0000 7f9ada453700 1 -- 172.21.15.134:0/3098503642 learned_addr learned my addr 172.21.15.134:0/3098503642 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9ada453700 1 -- 172.21.15.134:0/3098503642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4005040 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9ada453700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f9ac4014000 tx=0x7f9ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac40095f0 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad4106e20 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ac4007e70 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac4014ea0 con 0x7f9ad40f6510 2023-11-16T01:17:36.104 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.102+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41072b0 con 0x7f9ad40f6510 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.103+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ac40079c0 con 0x7f9ad40f6510 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.103+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4045ad0 con 0x7f9ad40f6510 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.103+0000 7f9acaffd700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 0x7f9ac0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.103+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ac4040860 con 0x7f9ad40f6510 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.103+0000 7f9ad9c52700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 0x7f9ac0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:36.105 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.104+0000 7f9ad9c52700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 0x7f9ac0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:36.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.108+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ac400f950 con 0x7f9ad40f6510 2023-11-16T01:17:36.311 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.310+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ad40fbf60 con 0x7f9ad40f6510 2023-11-16T01:17:36.312 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.311+0000 7f9acaffd700 1 -- 172.21.15.134:0/3098503642 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ac4010230 con 0x7f9ad40f6510 2023-11-16T01:17:36.312 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:36.312 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:36.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 msgr2=0x7f9ac0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:36.313 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 0x7f9ac0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).stop 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 msgr2=0x7f9ad4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f9ac4014000 tx=0x7f9ac400b040).stop 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 shutdown_connections 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f9ac002ff60 0x7f9ac0032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 --2- 172.21.15.134:0/3098503642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ad40f6510 0x7f9ad4106730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 >> 172.21.15.134:0/3098503642 conn(0x7f9ad40f1c70 msgr2=0x7f9ad40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 shutdown_connections 2023-11-16T01:17:36.314 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:36.313+0000 7f9adc6b7700 1 -- 172.21.15.134:0/3098503642 wait complete. 2023-11-16T01:17:36.315 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:37.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:36 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:36.311792+0000 mon.smithi133 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.134:0/3098503642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:37.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:37.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:38.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:37 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:36.017044+0000 mgr.smithi133.xdyttl (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:38.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 -- 172.21.15.134:0/2950365173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 msgr2=0x7fa2840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:38.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/2950365173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa2840f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa278009fb0 tx=0x7fa278009b30).stop 2023-11-16T01:17:38.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 -- 172.21.15.134:0/2950365173 shutdown_connections 2023-11-16T01:17:38.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/2950365173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa2840f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:38.895 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 -- 172.21.15.134:0/2950365173 >> 172.21.15.134:0/2950365173 conn(0x7fa2840f1c70 msgr2=0x7fa2840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 -- 172.21.15.134:0/2950365173 shutdown_connections 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.893+0000 7fa28b5d6700 1 -- 172.21.15.134:0/2950365173 wait complete. 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa28b5d6700 1 Processor -- start 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa28b5d6700 1 -- start start 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa28b5d6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa28b5d6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa284106c20 con 0x7fa2840f6510 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa289372700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa289372700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59794/0 (socket says 172.21.15.134:59794) 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa289372700 1 -- 172.21.15.134:0/3969478986 learned_addr learned my addr 172.21.15.134:0/3969478986 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa289372700 1 -- 172.21.15.134:0/3969478986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa278005040 con 0x7fa2840f6510 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa289372700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa278013510 tx=0x7fa27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:38.896 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa278009320 con 0x7fa2840f6510 2023-11-16T01:17:38.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa278013f00 con 0x7fa2840f6510 2023-11-16T01:17:38.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.894+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa284106e20 con 0x7fa2840f6510 2023-11-16T01:17:38.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.895+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa278014d90 con 0x7fa2840f6510 2023-11-16T01:17:38.897 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.895+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2841072b0 con 0x7fa2840f6510 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.895+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa278014420 con 0x7fa2840f6510 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.895+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa284045ad0 con 0x7fa2840f6510 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.896+0000 7fa275ffb700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 0x7fa270032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.896+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2780410c0 con 0x7fa2840f6510 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.896+0000 7fa288b71700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 0x7fa270032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:38.898 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.896+0000 7fa288b71700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 0x7fa270032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa2800096a0 tx=0x7fa280006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:38.901 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:38.900+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2780095f0 con 0x7fa2840f6510 2023-11-16T01:17:39.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.105+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2840fbf60 con 0x7fa2840f6510 2023-11-16T01:17:39.106 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.106+0000 7fa275ffb700 1 -- 172.21.15.134:0/3969478986 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa27801db70 con 0x7fa2840f6510 2023-11-16T01:17:39.107 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:39.107 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:39.108 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.107+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 msgr2=0x7fa270032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.107+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 0x7fa270032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa2800096a0 tx=0x7fa280006b40).stop 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.107+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 msgr2=0x7fa284106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.107+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa278013510 tx=0x7fa27800b040).stop 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 shutdown_connections 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa27002ff60 0x7fa270032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 --2- 172.21.15.134:0/3969478986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2840f6510 0x7fa284106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 >> 172.21.15.134:0/3969478986 conn(0x7fa2840f1c70 msgr2=0x7fa2840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:39.109 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 shutdown_connections 2023-11-16T01:17:39.110 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:39.108+0000 7fa28b5d6700 1 -- 172.21.15.134:0/3969478986 wait complete. 2023-11-16T01:17:39.110 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:38.017306+0000 mgr.smithi133.xdyttl (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:40.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:39 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:39.106275+0000 mon.smithi133 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.134:0/3969478986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:40.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:40.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 -- 172.21.15.134:0/1719966561 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 msgr2=0x7f04440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 --2- 172.21.15.134:0/1719966561 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f04440f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f0438009fb0 tx=0x7f0438009b30).stop 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 -- 172.21.15.134:0/1719966561 shutdown_connections 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 --2- 172.21.15.134:0/1719966561 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f04440f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 -- 172.21.15.134:0/1719966561 >> 172.21.15.134:0/1719966561 conn(0x7f04440f1c70 msgr2=0x7f04440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:41.693 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 -- 172.21.15.134:0/1719966561 shutdown_connections 2023-11-16T01:17:41.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.692+0000 7f044ae8d700 1 -- 172.21.15.134:0/1719966561 wait complete. 2023-11-16T01:17:41.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f044ae8d700 1 Processor -- start 2023-11-16T01:17:41.694 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f044ae8d700 1 -- start start 2023-11-16T01:17:41.695 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f044ae8d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f044ae8d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0444106c20 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f0448c29700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f0448c29700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59682/0 (socket says 172.21.15.134:59682) 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f0448c29700 1 -- 172.21.15.134:0/3181374729 learned_addr learned my addr 172.21.15.134:0/3181374729 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.693+0000 7f0448c29700 1 -- 172.21.15.134:0/3181374729 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438005040 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f0448c29700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f0438013510 tx=0x7f043800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0438009320 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444106e20 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0438013f00 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0438014d90 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04441072b0 con 0x7f04440f6510 2023-11-16T01:17:41.696 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.694+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0438014420 con 0x7f04440f6510 2023-11-16T01:17:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.695+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0444045ad0 con 0x7f04440f6510 2023-11-16T01:17:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.695+0000 7f04417fa700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 0x7f0434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.695+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04380410c0 con 0x7f04440f6510 2023-11-16T01:17:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.695+0000 7f0443fff700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 0x7f0434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.695+0000 7f0443fff700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 0x7f0434032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f04300096a0 tx=0x7f0430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:41.700 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.699+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04380095f0 con 0x7f04440f6510 2023-11-16T01:17:41.906 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.906+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04440fbf60 con 0x7f04440f6510 2023-11-16T01:17:41.907 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.907+0000 7f04417fa700 1 -- 172.21.15.134:0/3181374729 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f043801db70 con 0x7f04440f6510 2023-11-16T01:17:41.907 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:41.907 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 msgr2=0x7f0434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 0x7f0434032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f04300096a0 tx=0x7f0430006b40).stop 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 msgr2=0x7f0444106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f0438013510 tx=0x7f043800b040).stop 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 shutdown_connections 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7f043402ff60 0x7f0434032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 --2- 172.21.15.134:0/3181374729 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04440f6510 0x7f0444106730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 >> 172.21.15.134:0/3181374729 conn(0x7f04440f1c70 msgr2=0x7f04440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 shutdown_connections 2023-11-16T01:17:41.909 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:41.908+0000 7f044ae8d700 1 -- 172.21.15.134:0/3181374729 wait complete. 2023-11-16T01:17:41.910 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:42.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:41 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:40.017687+0000 mgr.smithi133.xdyttl (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:43.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:42 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:41.907208+0000 mon.smithi133 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.134:0/3181374729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:43.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:43.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:44.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:43 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:42.017983+0000 mgr.smithi133.xdyttl (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 -- 172.21.15.134:0/1350689326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 msgr2=0x7fcd740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 --2- 172.21.15.134:0/1350689326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd740f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fcd5c009fb0 tx=0x7fcd5c009b30).stop 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 -- 172.21.15.134:0/1350689326 shutdown_connections 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 --2- 172.21.15.134:0/1350689326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd740f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 -- 172.21.15.134:0/1350689326 >> 172.21.15.134:0/1350689326 conn(0x7fcd740f0b70 msgr2=0x7fcd740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 -- 172.21.15.134:0/1350689326 shutdown_connections 2023-11-16T01:17:44.527 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 -- 172.21.15.134:0/1350689326 wait complete. 2023-11-16T01:17:44.528 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.525+0000 7fcd7a897700 1 Processor -- start 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd7a897700 1 -- start start 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd7a897700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd7a897700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74105af0 con 0x7fcd740f5410 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd73fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd73fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59694/0 (socket says 172.21.15.134:59694) 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.526+0000 7fcd73fff700 1 -- 172.21.15.134:0/2342488706 learned_addr learned my addr 172.21.15.134:0/2342488706 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd73fff700 1 -- 172.21.15.134:0/2342488706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c005040 con 0x7fcd740f5410 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd73fff700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcd5c013510 tx=0x7fcd5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5c009320 con 0x7fcd740f5410 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd5c013f00 con 0x7fcd740f5410 2023-11-16T01:17:44.529 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd74105cf0 con 0x7fcd740f5410 2023-11-16T01:17:44.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd5c014d90 con 0x7fcd740f5410 2023-11-16T01:17:44.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.527+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd74106180 con 0x7fcd740f5410 2023-11-16T01:17:44.530 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd5c014420 con 0x7fcd740f5410 2023-11-16T01:17:44.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd74045ad0 con 0x7fcd740f5410 2023-11-16T01:17:44.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd70ff9700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 0x7fcd60032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:44.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd5c040ff0 con 0x7fcd740f5410 2023-11-16T01:17:44.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd737fe700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 0x7fcd60032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:44.531 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.528+0000 7fcd737fe700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 0x7fcd60032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd640096a0 tx=0x7fcd64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:44.534 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.532+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd5c06d2b0 con 0x7fcd740f5410 2023-11-16T01:17:44.739 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.737+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd740fae60 con 0x7fcd740f5410 2023-11-16T01:17:44.739 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.738+0000 7fcd70ff9700 1 -- 172.21.15.134:0/2342488706 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd5c0095f0 con 0x7fcd740f5410 2023-11-16T01:17:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:44.740 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:44.741 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.739+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 msgr2=0x7fcd60032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:44.741 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.739+0000 7fcd7a897700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 0x7fcd60032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd640096a0 tx=0x7fcd64006b40).stop 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.739+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 msgr2=0x7fcd74105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.739+0000 7fcd7a897700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcd5c013510 tx=0x7fcd5c00b040).stop 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 shutdown_connections 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fcd6002ffb0 0x7fcd60032470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 --2- 172.21.15.134:0/2342488706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd740f5410 0x7fcd74105600 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 >> 172.21.15.134:0/2342488706 conn(0x7fcd740f0b70 msgr2=0x7fcd740f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 shutdown_connections 2023-11-16T01:17:44.742 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:44.740+0000 7fcd7a897700 1 -- 172.21.15.134:0/2342488706 wait complete. 2023-11-16T01:17:44.743 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:45.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:44 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:44.739421+0000 mon.smithi133 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.134:0/2342488706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:46.031 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:45 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:44.018261+0000 mgr.smithi133.xdyttl (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:46.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-16T01:17:46.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T01:17:47.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 -- 172.21.15.134:0/406147982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 msgr2=0x7fa6e80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:47.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/406147982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e80f68a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fa6d0009fb0 tx=0x7fa6d0009b30).stop 2023-11-16T01:17:47.385 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 -- 172.21.15.134:0/406147982 shutdown_connections 2023-11-16T01:17:47.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/406147982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e80f68a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:47.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 -- 172.21.15.134:0/406147982 >> 172.21.15.134:0/406147982 conn(0x7fa6e80f1c00 msgr2=0x7fa6e80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:47.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 -- 172.21.15.134:0/406147982 shutdown_connections 2023-11-16T01:17:47.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.383+0000 7fa6ee25f700 1 -- 172.21.15.134:0/406147982 wait complete. 2023-11-16T01:17:47.386 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.384+0000 7fa6ee25f700 1 Processor -- start 2023-11-16T01:17:47.387 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.384+0000 7fa6ee25f700 1 -- start start 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.384+0000 7fa6ee25f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.384+0000 7fa6ee25f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8106bc0 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.134:59718/0 (socket says 172.21.15.134:59718) 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6e77fe700 1 -- 172.21.15.134:0/2190317240 learned_addr learned my addr 172.21.15.134:0/2190317240 (peer_addr_for_me v2:172.21.15.134:0/0) 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6e77fe700 1 -- 172.21.15.134:0/2190317240 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0005040 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6e77fe700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa6d0013510 tx=0x7fa6d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d0009320 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e8106dc0 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d0013f00 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d0014d90 con 0x7fa6e80f6480 2023-11-16T01:17:47.388 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.385+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e8107250 con 0x7fa6e80f6480 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.386+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6d0014420 con 0x7fa6e80f6480 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.386+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e8045ad0 con 0x7fa6e80f6480 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.386+0000 7fa6dffff700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.386+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6d0041070 con 0x7fa6e80f6480 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.386+0000 7fa6e6ffd700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-16T01:17:47.389 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.387+0000 7fa6e6ffd700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 0x7fa6d4032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa6d80096a0 tx=0x7fa6d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T01:17:47.392 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.391+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6d00095f0 con 0x7fa6e80f6480 2023-11-16T01:17:47.600 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.598+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6e8108100 con 0x7fa6e80f6480 2023-11-16T01:17:47.600 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.599+0000 7fa6dffff700 1 -- 172.21.15.134:0/2190317240 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa6d001db70 con 0x7fa6e80f6480 2023-11-16T01:17:47.600 INFO:teuthology.orchestra.run.smithi134.stdout: 2023-11-16T01:17:47.601 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"7905ec40-841c-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T01:07:36.340362Z","created":"2023-11-16T01:07:36.340362Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T01:17:47.602 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.600+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 msgr2=0x7fa6d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:47.602 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.600+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 0x7fa6d4032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa6d80096a0 tx=0x7fa6d8006b40).stop 2023-11-16T01:17:47.602 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 msgr2=0x7fa6e81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T01:17:47.602 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa6d0013510 tx=0x7fa6d000b040).stop 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 shutdown_connections 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:6800/2350789018,v1:172.21.15.133:6801/2350789018] conn(0x7fa6d402ff60 0x7fa6d4032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 --2- 172.21.15.134:0/2190317240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6e80f6480 0x7fa6e81066d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 >> 172.21.15.134:0/2190317240 conn(0x7fa6e80f1c00 msgr2=0x7fa6e80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 shutdown_connections 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:2023-11-16T01:17:47.601+0000 7fa6ee25f700 1 -- 172.21.15.134:0/2190317240 wait complete. 2023-11-16T01:17:47.603 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2023-11-16T01:17:48.027 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-16T01:17:48.028 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-16T01:17:48.028 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T01:17:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: cluster 2023-11-16T01:17:46.018617+0000 mgr.smithi133.xdyttl (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T01:17:48.030 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:47 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: audit 2023-11-16T01:17:47.600271+0000 mon.smithi133 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.134:0/2190317240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T01:17:48.052 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T01:17:48.071 INFO:tasks.cephadm:Stopping all daemons... 2023-11-16T01:17:48.071 INFO:tasks.cephadm.mon.smithi133:Stopping mon.smithi133... 2023-11-16T01:17:48.071 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133 2023-11-16T01:17:48.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:48 smithi133 systemd[1]: Stopping Ceph mon.smithi133 for 7905ec40-841c-11ee-8dbe-212e2dc638e7... 2023-11-16T01:17:48.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:17:48.209+0000 7f4d638fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi133 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-16T01:17:48.530 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Nov 16 01:17:48 smithi133 ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7-mon.smithi133[111776]: debug 2023-11-16T01:17:48.209+0000 7f4d638fd700 -1 mon.smithi133@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T01:17:48.723 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi133.service' 2023-11-16T01:17:48.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T01:17:48.825 INFO:tasks.cephadm.mon.smithi133:Stopped mon.smithi133 2023-11-16T01:17:48.825 INFO:tasks.cephadm.mon.smithi134:Stopping mon.smithi134... 2023-11-16T01:17:48.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134 2023-11-16T01:17:48.862 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134.service: Unit ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134.service not loaded. 2023-11-16T01:17:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-16T01:17:48.864 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi134" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134' 2023-11-16T01:17:48.865 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-16T01:17:48.868 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/ubuntu@smithi133.front.sepia.ceph.com/crash 2023-11-16T01:17:48.869 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T01:17:48.912 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/ubuntu@smithi134.front.sepia.ceph.com/crash 2023-11-16T01:17:48.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T01:17:48.966 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T01:17:48.966 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2023-11-16T01:17:48.967 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-16T01:17:48.967 DEBUG:teuthology.orchestra.run.smithi133:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-16T01:17:49.001 INFO:teuthology.orchestra.run.smithi133.stderr:grep: /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-16T01:17:49.002 INFO:tasks.cephadm:Compressing logs... 2023-11-16T01:17:49.002 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T01:17:49.045 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-16T01:17:49.071 INFO:teuthology.orchestra.run.smithi133.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T01:17:49.071 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-16T01:17:49.071 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-16T01:17:49.071 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi133.log 2023-11-16T01:17:49.071 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi133.xdyttl.log 2023-11-16T01:17:49.072 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi133.log: /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi133.xdyttl.log: 90.8% -- replaced with /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-16T01:17:49.075 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-16T01:17:49.075 INFO:teuthology.orchestra.run.smithi134.stderr:: No such file or directory 2023-11-16T01:17:49.077 INFO:teuthology.orchestra.run.smithi133.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-16T01:17:49.077 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-16T01:17:49.079 INFO:teuthology.orchestra.run.smithi134.stderr: 2023-11-16T01:17:49.079 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m0.014s 2023-11-16T01:17:49.079 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m0.011s 2023-11-16T01:17:49.079 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.012s 2023-11-16T01:17:49.149 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi133.xdyttl.log: file size changed while zipping 2023-11-16T01:17:49.149 INFO:teuthology.orchestra.run.smithi133.stderr: 92.7% -- replaced with /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi133.xdyttl.log.gz 2023-11-16T01:17:49.214 INFO:teuthology.orchestra.run.smithi133.stderr: 90.6% -- replaced with /var/log/ceph/7905ec40-841c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi133.log.gz 2023-11-16T01:17:49.216 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-11-16T01:17:49.216 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m0.155s 2023-11-16T01:17:49.216 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m0.232s 2023-11-16T01:17:49.216 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.017s 2023-11-16T01:17:49.216 INFO:tasks.cephadm:Archiving logs... 2023-11-16T01:17:49.216 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/ubuntu@smithi133.front.sepia.ceph.com/log 2023-11-16T01:17:49.217 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T01:17:49.338 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/ubuntu@smithi134.front.sepia.ceph.com/log 2023-11-16T01:17:49.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T01:17:49.414 INFO:tasks.cephadm:Removing cluster... 2023-11-16T01:17:49.414 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 --force 2023-11-16T01:18:23.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7905ec40-841c-11ee-8dbe-212e2dc638e7 --force 2023-11-16T01:18:23.953 INFO:tasks.cephadm:Removing cephadm ... 2023-11-16T01:18:23.953 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T01:18:23.972 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T01:18:23.991 INFO:tasks.cephadm:Teardown complete 2023-11-16T01:18:23.991 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134' 2023-11-16T01:18:24.095 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2109b3ae4d0244bdad2158692905393b Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 5: 'sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134' 2023-11-16T01:18:24.099 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-16T01:18:24.108 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T01:18:24.121 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T01:18:24.121 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T01:18:24.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T01:18:24.222 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T01:18:24.233 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T01:18:24.233 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:18:24.238 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T01:18:24.251 INFO:teuthology.orchestra.run.smithi133.stderr:bash: ntpq: command not found 2023-11-16T01:18:24.254 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2023-11-16T01:18:24.256 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T01:18:24.256 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-11-16T01:18:24.256 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2113us[+2113us] +/- 97ms 2023-11-16T01:18:24.256 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 23 -445us[ -434us] +/- 58ms 2023-11-16T01:18:24.257 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -891us[ -891us] +/- 78ms 2023-11-16T01:18:24.257 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:18:24.260 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T01:18:24.260 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2023-11-16T01:18:24.260 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +2071us[+2071us] +/- 97ms 2023-11-16T01:18:24.260 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -494us[ -491us] +/- 58ms 2023-11-16T01:18:24.260 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -910us[ -907us] +/- 77ms 2023-11-16T01:18:24.261 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T01:18:24.261 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T01:18:24.272 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T01:18:24.273 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T01:18:24.287 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T01:18:24.385 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T01:18:24.475 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T01:18:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:18:24.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T01:18:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:18:24.546 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T01:18:24.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T01:18:24.598 INFO:teuthology.task.internal:Duration was 1443.298896 seconds 2023-11-16T01:18:24.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T01:18:24.621 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T01:18:24.622 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T01:18:24.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T01:18:24.668 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T01:18:24.670 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T01:18:24.792 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T01:18:24.793 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2023-11-16T01:18:24.793 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T01:18:24.816 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2023-11-16T01:18:24.817 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T01:18:24.838 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T01:18:24.839 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T01:18:24.859 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T01:18:24.926 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T01:18:24.927 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T01:18:24.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T01:18:25.034 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T01:18:25.045 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T01:18:25.046 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T01:18:25.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T01:18:25.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T01:18:25.116 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:18:25.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:18:25.143 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2023-11-16T01:18:25.167 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2023-11-16T01:18:25.193 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:18:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:18:25.251 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T01:18:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T01:18:25.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T01:18:25.280 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T01:18:25.281 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/smithi133 2023-11-16T01:18:25.282 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T01:18:25.335 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611/remote/smithi134 2023-11-16T01:18:25.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T01:18:25.376 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T01:18:25.376 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T01:18:25.379 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T01:18:25.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T01:18:25.443 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T01:18:25.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T01:18:25.459 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T01:18:25.460 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T01:18:25.462 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T01:18:25.477 INFO:teuthology.orchestra.run.smithi133.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 01:18 /home/ubuntu/cephtest 2023-11-16T01:18:25.485 INFO:teuthology.orchestra.run.smithi134.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 01:18 /home/ubuntu/cephtest 2023-11-16T01:18:25.486 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T01:18:25.499 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T01:18:25.533 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T01:18:25.561 DEBUG:teuthology.nuke:shortname: smithi133 2023-11-16T01:18:25.561 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T01:18:25.583 DEBUG:teuthology.nuke:shortname: smithi134 2023-11-16T01:18:25.583 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T01:18:25.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 00:43:39.261038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyAg+yJz4Z2rYuRwd2DCKUgHJEtkIGtylF3x6Lt3xLtYxAsgDhPScARCgpz0UnjTxtTKtdDpRcIYSfSVmN5cDE='} 2023-11-16T01:18:25.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-15_21:10:46-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7459611', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 00:43:39.258931', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQFNFq4x0TNBsuEXvW3zbKTAuxecTw3dBAgWgBx8ef4Klg+XKt5bO+aFQ7FJZ07K4DQ8kvQtf8GOMVo0EjJgBM='} 2023-11-16T01:18:25.643 INFO:teuthology.orchestra.console.smithi133:Power off 2023-11-16T01:18:25.643 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T01:18:25.667 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2023-11-16T01:18:25.668 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:25.689 INFO:teuthology.orchestra.console.smithi134:Power off 2023-11-16T01:18:25.690 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T01:18:25.706 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-11-16T01:18:25.729 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2023-11-16T01:18:25.729 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:25.772 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2023-11-16T01:18:29.707 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:29.773 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:29.821 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-11-16T01:18:29.890 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2023-11-16T01:18:33.827 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:33.890 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:33.940 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2023-11-16T01:18:34.003 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2023-11-16T01:18:34.041 INFO:teuthology.orchestra.console.smithi133:Power off completed 2023-11-16T01:18:38.006 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T01:18:38.120 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2023-11-16T01:18:38.224 INFO:teuthology.orchestra.console.smithi134:Power off completed 2023-11-16T01:18:38.329 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1443.2988955974579 failure_reason: 'Command failed on smithi134 with status 5: ''sudo systemctl stop ceph-7905ec40-841c-11ee-8dbe-212e2dc638e7@mon.smithi134''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2109b3ae4d0244bdad2158692905393b status: fail success: false 2023-11-16T01:18:38.329 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T01:18:38.434 INFO:teuthology.run:FAIL