2023-11-15T03:51:47.087 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T03:51:47.089 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/orig.config.yaml 2023-11-15T03:51:47.141 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T03:51:47.220 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458314' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhiSHYR7Wh5NTjX7HrNo6Qc/9bNi9zP2X3IjyHBlNpavBG4Y0oDlHIAuk3Re+vufcV58zJ/wt1088Kio9w+63o= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBUOLvmVbHExAjWaJuVTri3zapXa5LX9p9N2JCvbVN49oMFYf7g6zXL61+zGzCR2ad0TJSm2KshAEB21zRZ1hw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T03:51:47.221 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T03:51:47.221 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T03:51:47.222 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T03:51:47.223 INFO:teuthology.task.internal:Checking packages... 2023-11-15T03:51:47.247 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T03:51:47.247 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T03:51:47.248 INFO:teuthology.packaging:ref: None 2023-11-15T03:51:47.248 INFO:teuthology.packaging:tag: None 2023-11-15T03:51:47.249 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T03:51:47.249 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T03:51:47.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T03:51:47.460 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T03:51:47.460 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T03:51:47.468 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T03:51:47.468 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T03:51:47.508 INFO:teuthology.task.internal:Saving configuration 2023-11-15T03:51:47.529 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T03:51:47.535 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T03:51:47.561 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 03:46:39.261530', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhiSHYR7Wh5NTjX7HrNo6Qc/9bNi9zP2X3IjyHBlNpavBG4Y0oDlHIAuk3Re+vufcV58zJ/wt1088Kio9w+63o='} 2023-11-15T03:51:47.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 03:46:39.260567', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBUOLvmVbHExAjWaJuVTri3zapXa5LX9p9N2JCvbVN49oMFYf7g6zXL61+zGzCR2ad0TJSm2KshAEB21zRZ1hw='} 2023-11-15T03:51:47.584 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T03:51:47.590 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T03:51:47.590 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T03:51:47.590 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T03:51:47.678 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68dc33fb80>, signals=[15]) 2023-11-15T03:51:47.680 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T03:51:47.685 INFO:teuthology.task.internal:Opening connections... 2023-11-15T03:51:47.685 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2023-11-15T03:51:47.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:51:47.755 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2023-11-15T03:51:47.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:51:47.824 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T03:51:47.831 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2023-11-15T03:51:47.855 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2023-11-15T03:51:47.856 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2023-11-15T03:51:47.918 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2023-11-15T03:51:47.918 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2023-11-15T03:51:47.919 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2023-11-15T03:51:47.919 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2023-11-15T03:51:47.919 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2023-11-15T03:51:47.919 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2023-11-15T03:51:47.919 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T03:51:47.920 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T03:51:47.921 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2023-11-15T03:51:47.943 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2023-11-15T03:51:47.965 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2023-11-15T03:51:47.966 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2023-11-15T03:51:48.026 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2023-11-15T03:51:48.026 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2023-11-15T03:51:48.026 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2023-11-15T03:51:48.027 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2023-11-15T03:51:48.027 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2023-11-15T03:51:48.027 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2023-11-15T03:51:48.027 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T03:51:48.027 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2023-11-15T03:51:48.028 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T03:51:48.028 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2023-11-15T03:51:48.028 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T03:51:48.028 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T03:51:48.028 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T03:51:48.029 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-11-15T03:51:48.050 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T03:51:48.059 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T03:51:48.091 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T03:51:48.091 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2023-11-15T03:51:48.094 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2023-11-15T03:51:48.114 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T03:51:48.119 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T03:51:48.119 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2023-11-15T03:51:48.156 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2023-11-15T03:51:48.180 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T03:51:48.232 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T03:51:48.248 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T03:51:48.249 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T03:51:48.250 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T03:51:48.250 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T03:51:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T03:51:48.271 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2023-11-15T03:51:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T03:51:48.272 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2023-11-15T03:51:48.331 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-489.el8.x86_64 2023-11-15T03:51:48.331 INFO:teuthology.task.kernel:Running kernel on smithi188: 4.18.0-489.el8.x86_64 2023-11-15T03:51:48.332 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2023-11-15T03:51:48.333 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-489.el8.x86_64 2023-11-15T03:51:48.333 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-489.el8.x86_64 2023-11-15T03:51:48.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2023-11-15T03:51:52.570 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 21 MB/s | 34 MB 00:01 2023-11-15T03:51:52.605 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 19 MB/s | 34 MB 00:01 2023-11-15T03:51:59.871 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 47 MB/s | 53 MB 00:01 2023-11-15T03:52:00.337 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 46 MB/s | 53 MB 00:01 2023-11-15T03:52:08.229 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 22 kB/s | 18 kB 00:00 2023-11-15T03:52:08.861 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 36 kB/s | 18 kB 00:00 2023-11-15T03:52:08.896 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-11-15T03:52:09.189 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-11-15T03:52:09.545 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-11-15T03:52:09.872 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-15T03:52:11.992 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2023-11-15T03:52:16.106 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 506 kB/s | 24 kB 00:00 2023-11-15T03:52:19.898 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T03:52:19.898 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T03:52:20.201 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T03:52:20.202 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T03:52:20.202 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T03:52:20.202 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T03:52:20.203 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T03:52:20.203 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T03:52:20.203 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T03:52:20.203 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T03:52:20.203 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T03:52:20.204 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:52:20.204 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T03:52:20.204 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T03:52:20.204 INFO:teuthology.orchestra.run.smithi038.stdout:Install 3 Packages 2023-11-15T03:52:20.204 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:52:20.205 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 89 M 2023-11-15T03:52:20.205 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 96 M 2023-11-15T03:52:20.206 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T03:52:20.566 INFO:teuthology.orchestra.run.smithi038.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 31 MB/s | 10 MB 00:00 2023-11-15T03:52:21.191 INFO:teuthology.orchestra.run.smithi038.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 46 MB/s | 43 MB 00:00 2023-11-15T03:52:21.967 INFO:teuthology.orchestra.run.smithi038.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2023-11-15T03:52:21.967 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T03:52:21.967 INFO:teuthology.orchestra.run.smithi038.stdout:Total 51 MB/s | 89 MB 00:01 2023-11-15T03:52:22.674 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T03:52:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T03:52:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T03:52:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T03:52:24.474 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T03:52:25.414 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:16 2023-11-15T03:52:27.392 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T03:52:27.999 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:52:29.178 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 579 kB/s | 24 kB 00:00 2023-11-15T03:52:31.762 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:52:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:52:32.936 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T03:52:32.936 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T03:52:33.223 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T03:52:33.224 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T03:52:33.224 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T03:52:33.224 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T03:52:33.225 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T03:52:33.225 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T03:52:33.225 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T03:52:33.225 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T03:52:33.225 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T03:52:33.226 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:52:33.226 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T03:52:33.226 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T03:52:33.226 INFO:teuthology.orchestra.run.smithi188.stdout:Install 3 Packages 2023-11-15T03:52:33.226 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:52:33.227 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 89 M 2023-11-15T03:52:33.227 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 96 M 2023-11-15T03:52:33.227 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T03:52:33.754 INFO:teuthology.orchestra.run.smithi188.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 30 MB/s | 10 MB 00:00 2023-11-15T03:52:34.521 INFO:teuthology.orchestra.run.smithi188.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 39 MB/s | 43 MB 00:01 2023-11-15T03:52:34.846 INFO:teuthology.orchestra.run.smithi188.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 25 MB/s | 36 MB 00:01 2023-11-15T03:52:34.847 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T03:52:34.847 INFO:teuthology.orchestra.run.smithi188.stdout:Total 55 MB/s | 89 MB 00:01 2023-11-15T03:52:35.520 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T03:52:35.822 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:52:35.931 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T03:52:35.932 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T03:52:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:52:37.148 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T03:52:37.171 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T03:52:39.900 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T03:52:40.438 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:52:43.985 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:52:44.353 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:52:47.822 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:52:48.292 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:52:59.312 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:52:59.338 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:52:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:52:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:52:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:53:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:53:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:53:00.546 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T03:53:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T03:53:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T03:53:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:53:00.547 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T03:53:00.781 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T03:53:01.009 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T03:53:01.009 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T03:53:01.010 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T03:53:01.010 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T03:53:01.010 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T03:53:01.011 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T03:53:01.011 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T03:53:01.011 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T03:53:01.230 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T03:53:01.230 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T03:53:01.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2023-11-15T03:53:01.266 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2023-11-15T03:53:01.266 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2023-11-15T03:53:01.266 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.02 2023-11-15T03:53:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 129.el8 2023-11-15T03:53:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2023-11-15T03:53:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T03:53:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:Group : System Environment/Base 2023-11-15T03:53:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 9265586 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T03:53:01.268 INFO:teuthology.orchestra.run.smithi038.stdout:Relocations : (not relocatable) 2023-11-15T03:53:01.269 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : CentOS BuildSystem 2023-11-15T03:53:01.269 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2023-11-15T03:53:01.269 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T03:53:01.269 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2023-11-15T03:53:01.269 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2023-11-15T03:53:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:53:01.270 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T03:53:01.270 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T03:53:01.270 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T03:53:01.271 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2023-11-15T03:53:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T03:53:01.271 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T03:53:01.272 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T03:53:01.272 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 4.18.0-521.el8.x86_64 2023-11-15T03:53:01.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T03:53:02.634 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2023-11-15T03:53:05.006 INFO:teuthology.orchestra.run.smithi038.stderr:done 2023-11-15T03:53:05.007 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2023-11-15T03:53:05.034 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.HRsjllvRYL 2023-11-15T03:53:05.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HRsjllvRYL 2023-11-15T03:53:05.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.HRsjllvRYL 2023-11-15T03:53:05.261 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.HRsjllvRYL is 6KB 2023-11-15T03:53:05.309 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.HRsjllvRYL 2023-11-15T03:53:05.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T03:53:05.400 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T03:53:05.400 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T03:53:05.400 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T03:53:05.401 INFO:teuthology.orchestra.run.smithi038.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T03:53:05.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T03:53:05.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2023-11-15T03:53:10.355 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:53:10.378 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:53:10.717 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:53:10.717 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T03:53:10.717 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T03:53:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T03:53:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:53:11.501 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T03:53:11.501 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T03:53:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T03:53:11.502 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:53:11.502 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T03:53:11.652 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T03:53:11.865 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T03:53:11.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T03:53:11.866 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T03:53:11.867 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T03:53:11.867 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T03:53:11.867 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T03:53:11.867 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T03:53:11.867 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T03:53:12.075 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T03:53:12.076 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T03:53:12.076 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2023-11-15T03:53:12.110 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2023-11-15T03:53:12.110 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2023-11-15T03:53:12.110 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.02 2023-11-15T03:53:12.110 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 129.el8 2023-11-15T03:53:12.111 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2023-11-15T03:53:12.111 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T03:53:12.111 INFO:teuthology.orchestra.run.smithi188.stdout:Group : System Environment/Base 2023-11-15T03:53:12.111 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 9265586 2023-11-15T03:53:12.111 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Relocations : (not relocatable) 2023-11-15T03:53:12.112 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : CentOS BuildSystem 2023-11-15T03:53:12.113 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2023-11-15T03:53:12.113 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T03:53:12.113 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2023-11-15T03:53:12.113 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2023-11-15T03:53:12.113 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:53:12.114 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T03:53:12.114 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T03:53:12.114 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T03:53:12.114 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2023-11-15T03:53:12.114 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T03:53:12.115 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T03:53:12.116 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T03:53:12.116 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 4.18.0-521.el8.x86_64 2023-11-15T03:53:12.116 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T03:53:13.787 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2023-11-15T03:53:15.953 INFO:teuthology.orchestra.run.smithi188.stderr:done 2023-11-15T03:53:15.954 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2023-11-15T03:53:15.987 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.xzngyfrrhe 2023-11-15T03:53:15.987 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xzngyfrrhe 2023-11-15T03:53:16.061 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.xzngyfrrhe 2023-11-15T03:53:16.204 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.xzngyfrrhe is 6KB 2023-11-15T03:53:16.252 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.xzngyfrrhe 2023-11-15T03:53:16.274 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T03:53:16.338 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T03:53:16.339 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T03:53:16.339 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T03:53:16.339 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T03:53:16.340 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T03:53:16.498 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2023-11-15T03:53:35.603 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T03:53:35.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T03:53:35.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:53:46.510 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T03:53:46.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-15T03:53:46.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:54:10.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-11-15T03:54:19.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T03:54:19.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:54:22.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-11-15T03:54:34.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T03:54:34.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:54:46.522 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T03:54:55.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-15T03:54:55.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T03:54:55.946 DEBUG:teuthology.orchestra.run.smithi188:> true 2023-11-15T03:54:56.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-15T03:54:56.205 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T03:54:56.205 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2023-11-15T03:54:56.222 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-521.el8.x86_64 2023-11-15T03:54:56.223 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T03:54:56.223 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T03:54:56.223 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T03:54:57.225 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T03:54:57.225 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T03:54:57.307 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2023-11-15T03:54:57.321 DEBUG:teuthology.parallel:result is None 2023-11-15T03:55:06.095 DEBUG:teuthology.orchestra.run.smithi038:> true 2023-11-15T03:55:06.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T03:55:06.441 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T03:55:06.442 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2023-11-15T03:55:06.462 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-521.el8.x86_64 2023-11-15T03:55:06.462 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T03:55:06.463 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T03:55:06.463 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T03:55:07.464 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T03:55:07.464 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T03:55:07.610 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2023-11-15T03:55:07.626 DEBUG:teuthology.parallel:result is None 2023-11-15T03:55:07.627 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T03:55:07.634 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T03:55:07.634 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T03:55:07.637 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T03:55:07.660 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T03:55:07.676 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T03:55:07.726 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T03:55:07.726 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T03:55:07.729 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T03:55:07.765 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T03:55:07.776 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T03:55:07.776 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T03:55:07.794 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T03:55:07.837 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T03:55:07.846 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T03:55:07.850 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T03:55:07.857 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T03:55:07.859 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T03:55:07.865 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T03:55:07.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T03:55:07.868 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T03:55:07.898 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T03:55:07.936 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T03:55:07.937 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T03:55:07.939 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T03:55:07.958 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T03:55:08.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T03:55:08.094 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T03:55:08.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T03:55:08.229 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T03:55:08.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T03:55:08.299 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T03:55:08.347 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T03:55:08.405 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T03:55:08.492 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T03:55:08.536 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T03:55:08.536 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T03:55:08.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2023-11-15T03:55:08.609 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2023-11-15T03:55:08.675 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T03:55:08.711 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T03:55:09.182 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T03:55:09.189 INFO:teuthology.task.internal:Starting timer... 2023-11-15T03:55:09.190 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T03:55:09.231 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T03:55:09.279 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T03:55:09.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2023-11-15T03:55:09.376 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2023-11-15T03:55:09.378 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2023-11-15T03:55:09.452 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2023-11-15T03:55:09.454 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T03:55:09.455 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2023-11-15T03:55:09.492 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2023-11-15T03:55:09.492 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2023-11-15T03:55:09.541 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2023-11-15T03:55:09.542 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2023-11-15T03:55:09.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T03:55:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T03:55:09.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T03:55:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T03:55:09.607 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T03:55:09.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2023-11-15T03:55:09.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2023-11-15T03:55:09.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T03:55:09.705 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T03:55:09.706 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T03:55:09.727 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T03:55:09.729 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T04:03:31.094 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2023-11-15T04:03:31.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T04:03:31.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:03:31.174 DEBUG:teuthology.orchestra.run.smithi038:> true 2023-11-15T04:03:31.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-15T04:03:31.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-15T04:03:31.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:03:31.320 DEBUG:teuthology.orchestra.run.smithi188:> true 2023-11-15T04:03:31.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-11-15T04:03:31.389 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T04:03:31.405 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T04:03:31.405 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:03:31.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:03:31.409 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:03:31.409 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:03:31.447 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:03:31.461 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:03:31.464 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:03:31.475 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:03:31.530 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2023-11-15T04:03:31.534 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2023-11-15T04:03:31.542 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2023-11-15T04:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2023-11-15T04:03:31.556 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:03:31.564 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:03:31.569 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:03:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:03:31.647 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2023-11-15T04:03:31.648 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-11-15T04:03:31.651 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:03:31.652 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2023-11-15T04:03:31.652 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.652 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.652 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.652 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.654 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:03:31.654 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-11-15T04:03:31.654 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.655 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.655 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.655 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:03:31.655 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:03:31.670 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:03:31.671 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi038.front.sepia.ceph.com 2023-11-15T04:03:31.672 DEBUG:teuthology.orchestra.run.smithi038:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:03:31.672 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2023-11-15T04:03:31.672 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:03:32.576 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 Nov 2023 04:01:09 AM UTC. 2023-11-15T04:03:32.716 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Wed 15 Nov 2023 04:01:48 AM UTC. 2023-11-15T04:03:34.682 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:03:34.683 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:34.683 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:03:34.684 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:34.684 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2023-11-15T04:03:34.684 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2023-11-15T04:03:34.684 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:34.684 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:03:34.685 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:34.685 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:34.935 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:03:35.099 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:03:35.100 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:03:35.100 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:35.101 INFO:teuthology.orchestra.run.smithi038.stdout:Resetting modules: 2023-11-15T04:03:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools 2023-11-15T04:03:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:35.101 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:03:35.101 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:35.385 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:03:35.884 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:26 ago on Wed 15 Nov 2023 04:01:09 AM UTC. 2023-11-15T04:03:36.464 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 Nov 2023 04:01:48 AM UTC. 2023-11-15T04:03:37.998 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:03:38.176 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:38.176 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:03:38.176 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:38.177 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2023-11-15T04:03:38.177 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:03:38.178 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:03:38.178 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:03:38.178 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:03:38.179 INFO:teuthology.orchestra.run.smithi188.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:03:38.179 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:03:38.179 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:03:38.180 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:03:38.180 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:03:38.180 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:03:38.181 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:03:38.181 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:03:38.182 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:03:38.182 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2023-11-15T04:03:38.182 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:03:38.183 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2023-11-15T04:03:38.183 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2023-11-15T04:03:38.184 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:38.184 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2023-11-15T04:03:38.184 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2023-11-15T04:03:38.185 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:38.185 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:03:38.185 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:03:38.186 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2023-11-15T04:03:38.186 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:38.187 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 27 M 2023-11-15T04:03:38.188 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 89 M 2023-11-15T04:03:38.188 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:03:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:03:38.828 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:03:38.862 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.0 MB/s | 1.0 MB 00:00 2023-11-15T04:03:38.904 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 502 kB/s | 38 kB 00:00 2023-11-15T04:03:38.938 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2023-11-15T04:03:39.005 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2023-11-15T04:03:39.014 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:03:39.015 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:39.015 INFO:teuthology.orchestra.run.smithi038.stdout:Installing group/module packages: 2023-11-15T04:03:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:03:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:03:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:03:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:03:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:03:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:03:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:03:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:03:39.018 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2023-11-15T04:03:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:03:39.019 INFO:teuthology.orchestra.run.smithi038.stdout:Installing module profiles: 2023-11-15T04:03:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools/common 2023-11-15T04:03:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:39.019 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling module streams: 2023-11-15T04:03:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools rhel8 2023-11-15T04:03:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:39.020 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:03:39.020 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:03:39.020 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2023-11-15T04:03:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:39.021 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 27 M 2023-11-15T04:03:39.021 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 89 M 2023-11-15T04:03:39.021 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:03:39.097 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 270 kB/s | 25 kB 00:00 2023-11-15T04:03:39.206 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-15T04:03:39.231 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 702 kB/s | 94 kB 00:00 2023-11-15T04:03:39.289 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 10 MB/s | 3.9 MB 00:00 2023-11-15T04:03:39.365 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 701 kB/s | 53 kB 00:00 2023-11-15T04:03:39.423 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2023-11-15T04:03:39.449 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 490 kB/s | 41 kB 00:00 2023-11-15T04:03:39.529 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:03:39.563 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.0 MB/s | 1.0 MB 00:00 2023-11-15T04:03:39.605 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 505 kB/s | 38 kB 00:00 2023-11-15T04:03:39.638 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2023-11-15T04:03:39.649 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 18 MB/s | 8.1 MB 00:00 2023-11-15T04:03:39.650 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:03:39.650 INFO:teuthology.orchestra.run.smithi188.stdout:Total 18 MB/s | 27 MB 00:01 2023-11-15T04:03:39.722 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2023-11-15T04:03:39.798 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 331 kB/s | 25 kB 00:00 2023-11-15T04:03:39.921 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:03:39.925 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 11 MB/s | 9.6 MB 00:00 2023-11-15T04:03:39.943 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:03:39.944 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:03:39.948 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 624 kB/s | 94 kB 00:00 2023-11-15T04:03:40.015 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.6 MB/s | 3.9 MB 00:00 2023-11-15T04:03:40.082 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 792 kB/s | 53 kB 00:00 2023-11-15T04:03:40.141 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2023-11-15T04:03:40.158 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 547 kB/s | 41 kB 00:00 2023-11-15T04:03:40.193 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:03:40.195 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:03:40.299 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 22 MB/s | 8.1 MB 00:00 2023-11-15T04:03:40.300 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:03:40.300 INFO:teuthology.orchestra.run.smithi038.stdout:Total 21 MB/s | 27 MB 00:01 2023-11-15T04:03:40.597 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:03:40.621 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:03:40.621 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:03:40.883 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:03:40.910 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:03:40.913 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:03:41.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:03:41.289 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:03:41.654 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:03:41.859 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:03:41.862 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:03:42.060 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:03:42.095 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:03:42.251 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:03:42.504 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:03:42.629 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:03:43.117 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:03:43.613 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:03:43.866 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:03:44.096 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:03:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:03:44.359 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:03:44.378 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:03:44.601 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:03:44.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:03:44.873 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:03:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:03:45.630 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:03:45.877 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:03:46.087 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:03:46.225 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:03:46.449 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:03:46.492 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:03:46.492 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:03:46.492 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:03:46.492 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:03:46.493 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:03:46.493 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:03:46.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:03:46.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:03:46.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:03:46.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:03:46.494 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:03:46.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:03:46.495 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:03:46.658 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:03:47.140 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:03:47.140 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:03:47.141 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:03:47.141 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:03:47.141 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:03:47.141 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:03:47.141 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:03:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:03:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:03:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:03:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:03:47.142 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:03:47.143 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:03:47.453 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:03:47.454 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:47.454 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:03:47.454 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.454 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.455 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:47.455 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:03:47.455 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.455 INFO:teuthology.orchestra.run.smithi188.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.455 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:47.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:03:47.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:03:47.456 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.456 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:47.457 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:47.457 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:03:47.457 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:03:47.457 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:03:47.754 DEBUG:teuthology.parallel:result is None 2023-11-15T04:03:48.098 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:03:48.098 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:48.098 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:03:48.098 INFO:teuthology.orchestra.run.smithi038.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.099 INFO:teuthology.orchestra.run.smithi038.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.099 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:48.099 INFO:teuthology.orchestra.run.smithi038.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:03:48.099 INFO:teuthology.orchestra.run.smithi038.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.099 INFO:teuthology.orchestra.run.smithi038.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:48.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:03:48.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:03:48.100 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.100 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:03:48.101 INFO:teuthology.orchestra.run.smithi038.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:03:48.101 INFO:teuthology.orchestra.run.smithi038.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:03:48.101 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:03:48.101 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:03:48.378 DEBUG:teuthology.parallel:result is None 2023-11-15T04:03:48.378 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:03:48.387 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:03:48.387 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi038.front.sepia.ceph.com 2023-11-15T04:03:48.388 DEBUG:teuthology.orchestra.run.smithi038:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:03:48.388 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2023-11-15T04:03:48.388 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:03:48.548 DEBUG:teuthology.parallel:result is None 2023-11-15T04:03:48.565 DEBUG:teuthology.parallel:result is None 2023-11-15T04:03:48.566 INFO:teuthology.run_tasks:Running task install... 2023-11-15T04:03:48.578 DEBUG:teuthology.task.install:project ceph 2023-11-15T04:03:48.578 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T04:03:48.578 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:03:48.579 INFO:teuthology.task.install:Using flavor: default 2023-11-15T04:03:48.585 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T04:03:48.585 INFO:teuthology.task.install:extra packages: [] 2023-11-15T04:03:48.586 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:03:48.586 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:03:48.586 INFO:teuthology.packaging:ref: None 2023-11-15T04:03:48.587 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:03:48.587 INFO:teuthology.packaging:branch: None 2023-11-15T04:03:48.587 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:03:48.795 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:03:48.796 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:03:48.800 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:03:48.800 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:03:48.800 INFO:teuthology.packaging:ref: None 2023-11-15T04:03:48.801 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:03:48.801 INFO:teuthology.packaging:branch: None 2023-11-15T04:03:48.801 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:03:48.997 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:03:48.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:03:49.146 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:03:49.147 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:03:49.181 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:03:49.181 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:03:49.182 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:03:49.212 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:03:49.212 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:03:49.213 INFO:teuthology.packaging:ref: None 2023-11-15T04:03:49.213 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:03:49.213 INFO:teuthology.packaging:branch: None 2023-11-15T04:03:49.213 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:03:49.214 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:03:49.223 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:03:49.224 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:03:49.255 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:03:49.256 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:03:49.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:03:49.283 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:03:49.294 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:03:49.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:03:49.294 INFO:teuthology.packaging:ref: None 2023-11-15T04:03:49.295 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:03:49.295 INFO:teuthology.packaging:branch: None 2023-11-15T04:03:49.295 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:03:49.295 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:03:49.360 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:03:49.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:03:49.426 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2023-11-15T04:03:49.427 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2023-11-15T04:03:49.459 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:03:49.491 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2023-11-15T04:03:49.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2023-11-15T04:03:49.699 INFO:teuthology.orchestra.run.smithi188.stdout:76 files removed 2023-11-15T04:03:49.727 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2023-11-15T04:03:49.799 INFO:teuthology.orchestra.run.smithi038.stdout:76 files removed 2023-11-15T04:03:49.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2023-11-15T04:03:50.258 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 610 kB/s | 74 kB 00:00 2023-11-15T04:03:50.376 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 190 kB/s | 15 kB 00:00 2023-11-15T04:03:50.383 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 956 kB/s | 74 kB 00:00 2023-11-15T04:03:50.485 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2023-11-15T04:03:50.517 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 167 kB/s | 15 kB 00:00 2023-11-15T04:03:50.627 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2023-11-15T04:03:52.285 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 19 MB/s | 34 MB 00:01 2023-11-15T04:03:52.368 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-15T04:03:59.886 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 43 MB/s | 53 MB 00:01 2023-11-15T04:04:00.586 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - BaseOS 40 MB/s | 53 MB 00:01 2023-11-15T04:04:08.243 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 29 kB/s | 18 kB 00:00 2023-11-15T04:04:08.927 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-11-15T04:04:09.510 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2023-11-15T04:04:09.695 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.6 MB 00:00 2023-11-15T04:04:10.069 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-11-15T04:04:10.731 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.6 MB 00:00 2023-11-15T04:04:11.146 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-15T04:04:12.239 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-11-15T04:04:12.305 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-15T04:04:14.459 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2023-11-15T04:04:16.164 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 414 kB/s | 24 kB 00:00 2023-11-15T04:04:18.757 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 454 kB/s | 24 kB 00:00 2023-11-15T04:04:20.844 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:04:20.847 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:04:20.847 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:04:20.848 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:04:20.848 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:04:20.848 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:04:20.848 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:04:20.849 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:04:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:04:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:04:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:04:20.850 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:04:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:04:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:04:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:04:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:04:20.851 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:04:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:04:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:04:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:04:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:04:20.852 INFO:teuthology.orchestra.run.smithi188.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:04:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:04:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:04:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:04:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:04:20.853 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:04:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:04:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:04:20.854 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:04:20.854 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:04:20.854 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:04:20.855 INFO:teuthology.orchestra.run.smithi188.stdout:Install 25 Packages 2023-11-15T04:04:20.855 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2023-11-15T04:04:20.855 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:04:20.856 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 59 M 2023-11-15T04:04:20.856 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:04:21.235 INFO:teuthology.orchestra.run.smithi188.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2023-11-15T04:04:21.266 INFO:teuthology.orchestra.run.smithi188.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 832 kB/s | 25 kB 00:00 2023-11-15T04:04:21.308 INFO:teuthology.orchestra.run.smithi188.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T04:04:21.350 INFO:teuthology.orchestra.run.smithi188.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T04:04:21.450 INFO:teuthology.orchestra.run.smithi188.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-11-15T04:04:21.475 INFO:teuthology.orchestra.run.smithi188.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2023-11-15T04:04:21.501 INFO:teuthology.orchestra.run.smithi188.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:04:21.552 INFO:teuthology.orchestra.run.smithi188.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 213 kB 00:00 2023-11-15T04:04:21.585 INFO:teuthology.orchestra.run.smithi188.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T04:04:21.610 INFO:teuthology.orchestra.run.smithi188.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-11-15T04:04:21.635 INFO:teuthology.orchestra.run.smithi188.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T04:04:21.777 INFO:teuthology.orchestra.run.smithi188.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T04:04:22.053 INFO:teuthology.orchestra.run.smithi188.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-11-15T04:04:22.105 INFO:teuthology.orchestra.run.smithi188.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 309 kB/s | 145 kB 00:00 2023-11-15T04:04:22.187 INFO:teuthology.orchestra.run.smithi188.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 539 kB/s | 44 kB 00:00 2023-11-15T04:04:22.278 INFO:teuthology.orchestra.run.smithi188.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 461 kB/s | 42 kB 00:00 2023-11-15T04:04:22.321 INFO:teuthology.orchestra.run.smithi188.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-11-15T04:04:22.392 INFO:teuthology.orchestra.run.smithi188.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 576 kB/s | 354 kB 00:00 2023-11-15T04:04:22.421 INFO:teuthology.orchestra.run.smithi188.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 705 kB/s | 259 kB 00:00 2023-11-15T04:04:22.447 INFO:teuthology.orchestra.run.smithi188.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 638 kB/s | 80 kB 00:00 2023-11-15T04:04:22.622 INFO:teuthology.orchestra.run.smithi188.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 507 kB/s | 116 kB 00:00 2023-11-15T04:04:22.680 INFO:teuthology.orchestra.run.smithi188.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-11-15T04:04:22.714 INFO:teuthology.orchestra.run.smithi188.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 677 kB/s | 181 kB 00:00 2023-11-15T04:04:22.739 INFO:teuthology.orchestra.run.smithi188.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 502 kB/s | 59 kB 00:00 2023-11-15T04:04:22.765 INFO:teuthology.orchestra.run.smithi188.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 895 kB/s | 75 kB 00:00 2023-11-15T04:04:22.848 INFO:teuthology.orchestra.run.smithi188.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2023-11-15T04:04:22.907 INFO:teuthology.orchestra.run.smithi188.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2023-11-15T04:04:22.909 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:04:22.910 INFO:teuthology.orchestra.run.smithi188.stdout:Total 28 MB/s | 59 MB 00:02 2023-11-15T04:04:22.976 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:04:23.025 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:04:23.025 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:04:23.366 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:04:23.369 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:04:23.823 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:04:23.826 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:04:23.827 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:04:23.827 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:04:23.827 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:04:23.827 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:04:23.827 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2023-11-15T04:04:23.828 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:04:23.828 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:04:23.828 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:04:23.828 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:04:23.828 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:04:23.829 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:04:23.829 INFO:teuthology.orchestra.run.smithi038.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:04:23.829 INFO:teuthology.orchestra.run.smithi038.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:04:23.829 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:04:23.829 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:04:23.830 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:04:23.830 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:04:23.830 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:04:23.830 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:04:23.831 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:04:23.831 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:04:23.832 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:04:23.832 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:04:23.832 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:04:23.832 INFO:teuthology.orchestra.run.smithi038.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:04:23.832 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:04:23.833 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:04:23.834 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:04:23.834 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:04:23.834 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:04:23.834 INFO:teuthology.orchestra.run.smithi038.stdout:Install 25 Packages 2023-11-15T04:04:23.834 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2023-11-15T04:04:23.835 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:04:23.835 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 59 M 2023-11-15T04:04:23.836 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:04:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 31 MB/s | 6.0 MB 00:00 2023-11-15T04:04:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 829 kB/s | 25 kB 00:00 2023-11-15T04:04:24.359 INFO:teuthology.orchestra.run.smithi038.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 16 MB/s | 801 kB 00:00 2023-11-15T04:04:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T04:04:24.509 INFO:teuthology.orchestra.run.smithi038.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2023-11-15T04:04:24.551 INFO:teuthology.orchestra.run.smithi038.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.0 MB/s | 44 kB 00:00 2023-11-15T04:04:24.552 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:04:24.577 INFO:teuthology.orchestra.run.smithi038.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:04:24.602 INFO:teuthology.orchestra.run.smithi038.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T04:04:24.661 INFO:teuthology.orchestra.run.smithi038.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 6.5 MB/s | 387 kB 00:00 2023-11-15T04:04:24.694 INFO:teuthology.orchestra.run.smithi038.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T04:04:24.736 INFO:teuthology.orchestra.run.smithi038.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 114 kB 00:00 2023-11-15T04:04:24.789 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:04:24.878 INFO:teuthology.orchestra.run.smithi038.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2023-11-15T04:04:25.026 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:04:25.170 INFO:teuthology.orchestra.run.smithi038.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2023-11-15T04:04:25.179 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi038.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2023-11-15T04:04:25.288 INFO:teuthology.orchestra.run.smithi038.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 866 kB/s | 354 kB 00:00 2023-11-15T04:04:25.313 INFO:teuthology.orchestra.run.smithi038.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 376 kB/s | 44 kB 00:00 2023-11-15T04:04:25.405 INFO:teuthology.orchestra.run.smithi038.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 362 kB/s | 42 kB 00:00 2023-11-15T04:04:25.430 INFO:teuthology.orchestra.run.smithi038.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 402 kB/s | 47 kB 00:00 2023-11-15T04:04:25.489 INFO:teuthology.orchestra.run.smithi038.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2023-11-15T04:04:25.514 INFO:teuthology.orchestra.run.smithi038.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 757 kB/s | 259 kB 00:00 2023-11-15T04:04:25.574 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:04:25.722 INFO:teuthology.orchestra.run.smithi038.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 398 kB/s | 116 kB 00:00 2023-11-15T04:04:25.754 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:04:25.865 INFO:teuthology.orchestra.run.smithi038.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 516 kB/s | 181 kB 00:00 2023-11-15T04:04:25.882 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:04:25.890 INFO:teuthology.orchestra.run.smithi038.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 352 kB/s | 59 kB 00:00 2023-11-15T04:04:25.915 INFO:teuthology.orchestra.run.smithi038.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 719 kB/s | 306 kB 00:00 2023-11-15T04:04:25.940 INFO:teuthology.orchestra.run.smithi038.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 995 kB/s | 75 kB 00:00 2023-11-15T04:04:26.032 INFO:teuthology.orchestra.run.smithi038.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2023-11-15T04:04:26.116 INFO:teuthology.orchestra.run.smithi038.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2023-11-15T04:04:26.118 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:04:26.119 INFO:teuthology.orchestra.run.smithi038.stdout:Total 26 MB/s | 59 MB 00:02 2023-11-15T04:04:26.128 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:04:26.192 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:04:26.246 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:04:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:04:26.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:04:26.353 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:04:26.551 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:04:26.643 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:04:26.645 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:04:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:04:27.170 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:04:27.328 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:04:27.427 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:04:27.617 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:04:27.770 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:04:27.834 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:04:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:04:28.034 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:04:28.072 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:04:28.212 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:04:28.297 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:04:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:04:28.439 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:04:28.460 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:04:28.603 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:04:28.935 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:04:29.046 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:04:29.135 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:04:29.211 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:04:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:04:29.318 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:04:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:04:29.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:04:29.685 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:04:29.755 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:04:29.889 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:04:29.946 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:04:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:04:30.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:04:30.157 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:04:30.380 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:04:30.570 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:04:30.609 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:04:30.741 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:04:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:04:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:04:30.953 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:04:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:04:31.493 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:04:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:04:31.726 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:04:31.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:04:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:04:32.197 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:04:32.343 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:04:32.822 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:04:33.010 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:04:33.123 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:04:33.324 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:04:33.525 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:04:33.601 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:33.668 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:04:33.741 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:04:33.828 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:33.990 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:04:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:04:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:04:34.522 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:04:34.856 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:04:34.962 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:04:35.201 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:04:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:04:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:04:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:04:53.841 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:04:53.841 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2023-11-15T04:04:53.842 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2023-11-15T04:04:53.842 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-11-15T04:04:53.842 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2023-11-15T04:04:53.842 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2023-11-15T04:04:53.842 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2023-11-15T04:04:53.843 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2023-11-15T04:04:53.843 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2023-11-15T04:04:53.843 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-11-15T04:04:53.843 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:04:55.160 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:04:55.234 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:04:55.234 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:04:55.449 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:04:55.450 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:04:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:04:56.166 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:04:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:04:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:04:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:04:56.167 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:04:56.168 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:04:56.168 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:04:56.168 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:04:56.168 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:04:56.168 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:04:56.169 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:04:56.169 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:04:56.169 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:04:56.169 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:04:56.169 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:04:56.170 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:04:56.170 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:04:56.170 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:04:56.171 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:04:56.171 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:04:56.171 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:04:56.171 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:04:56.171 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:04:56.172 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:04:56.172 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:04:56.172 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:04:56.172 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:04:56.172 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:04:57.508 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:04:57.508 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:04:57.508 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2023-11-15T04:04:57.509 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.509 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:04:57.509 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.509 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.509 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.510 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.510 INFO:teuthology.orchestra.run.smithi188.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:04:57.510 INFO:teuthology.orchestra.run.smithi188.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:04:57.510 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:04:57.510 INFO:teuthology.orchestra.run.smithi188.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:04:57.511 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.511 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:04:57.511 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:04:57.511 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:04:57.511 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:04:57.512 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.513 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.513 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.513 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:04:57.513 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.513 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.514 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:04:57.514 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:04:57.514 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:04:57.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2023-11-15T04:04:58.693 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:42 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:00.259 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:05:00.259 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2023-11-15T04:05:00.259 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2023-11-15T04:05:00.259 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2023-11-15T04:05:00.260 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /run 2023-11-15T04:05:00.260 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2023-11-15T04:05:00.260 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2023-11-15T04:05:00.260 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2023-11-15T04:05:00.260 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2023-11-15T04:05:00.261 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2023-11-15T04:05:00.261 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:00.360 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:00.361 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:00.361 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:05:00.361 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:00.362 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:05:00.362 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:05:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:05:00.362 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:05:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:05:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:05:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:05:00.363 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:00.363 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:00.364 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:00.364 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2023-11-15T04:05:00.364 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:00.364 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 53 M 2023-11-15T04:05:00.365 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 229 M 2023-11-15T04:05:00.365 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 656 kB/s | 169 kB 00:00 2023-11-15T04:05:01.143 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2023-11-15T04:05:01.169 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 265 kB/s | 203 kB 00:00 2023-11-15T04:05:01.252 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 848 kB/s | 71 kB 00:00 2023-11-15T04:05:01.435 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 242 kB/s | 187 kB 00:00 2023-11-15T04:05:01.535 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 824 kB/s | 323 kB 00:00 2023-11-15T04:05:01.536 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:01.537 INFO:teuthology.orchestra.run.smithi188.stdout:Total 45 MB/s | 53 MB 00:01 2023-11-15T04:05:01.580 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:01.601 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:01.601 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:01.731 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:05:01.814 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:05:01.814 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:05:01.912 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:01.915 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:05:02.419 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:02.665 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:05:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:05:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:05:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:05:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:05:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:05:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:05:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:05:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:05:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:05:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:05:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:05:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:05:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:05:02.799 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:05:02.799 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:05:02.799 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:05:02.799 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:05:02.799 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:05:02.800 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:05:02.800 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:05:02.800 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:05:02.800 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:05:02.800 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:05:02.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:05:02.872 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:05:03.029 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:05:03.107 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:05:03.445 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:05:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:05:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:04.114 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2023-11-15T04:05:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.115 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:05:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:05:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:05:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:05:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:05:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:05:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:05:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:05:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:05:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:05:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:05:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:05:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:04.124 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:04.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2023-11-15T04:05:05.382 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:47 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:07.273 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:07.274 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:05:07.275 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:07.275 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:05:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:05:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:05:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:07.277 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:05:07.277 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:07.277 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2023-11-15T04:05:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:07.278 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 53 M 2023-11-15T04:05:07.278 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 229 M 2023-11-15T04:05:07.279 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:05:07.368 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2023-11-15T04:05:08.115 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2023-11-15T04:05:08.143 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 246 kB/s | 203 kB 00:00 2023-11-15T04:05:08.169 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 234 kB/s | 187 kB 00:00 2023-11-15T04:05:08.252 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.3 MB/s | 323 kB 00:00 2023-11-15T04:05:08.315 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:05:08.402 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 273 kB/s | 71 kB 00:00 2023-11-15T04:05:08.403 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:08.404 INFO:teuthology.orchestra.run.smithi038.stdout:Total 47 MB/s | 53 MB 00:01 2023-11-15T04:05:08.453 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:05:08.474 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:05:08.474 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:05:08.559 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:05:08.813 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:05:08.817 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:05:08.863 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:05:08.863 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:05:08.863 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:05:08.864 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:05:08.864 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:05:08.864 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:05:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:05:09.578 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:05:09.578 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:09.579 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:05:09.579 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:05:09.579 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:05:09.580 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:05:09.580 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:09.580 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:05:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:05:09.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2023-11-15T04:05:09.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:05:10.222 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:05:10.379 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:05:10.715 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:54 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:12.391 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:12.394 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:12.395 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:05:12.395 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:12.396 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:12.396 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:05:12.396 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:05:12.397 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:05:12.397 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:05:12.397 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:05:12.398 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:05:12.398 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:05:12.398 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:05:12.398 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:05:12.399 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:05:12.399 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:05:12.399 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:05:12.399 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:05:12.399 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:05:12.400 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:05:12.400 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:05:12.400 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:05:12.400 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:05:12.400 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:05:12.401 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:05:12.401 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:05:12.401 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:05:12.401 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:05:12.401 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:05:12.402 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:05:12.402 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:05:12.402 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:05:12.402 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:05:12.402 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:05:12.403 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:05:12.403 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:05:12.403 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:05:12.403 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:05:12.403 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:12.404 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:12.404 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:12.404 INFO:teuthology.orchestra.run.smithi188.stdout:Install 32 Packages 2023-11-15T04:05:12.404 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:12.405 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 M 2023-11-15T04:05:12.405 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 111 M 2023-11-15T04:05:12.406 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:12.746 INFO:teuthology.orchestra.run.smithi188.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 133 kB/s | 6.3 kB 00:00 2023-11-15T04:05:12.796 INFO:teuthology.orchestra.run.smithi188.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-15T04:05:12.838 INFO:teuthology.orchestra.run.smithi188.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2023-11-15T04:05:12.863 INFO:teuthology.orchestra.run.smithi188.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-11-15T04:05:12.980 INFO:teuthology.orchestra.run.smithi188.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-11-15T04:05:13.237 INFO:teuthology.orchestra.run.smithi188.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 41 MB/s | 18 MB 00:00 2023-11-15T04:05:13.273 INFO:teuthology.orchestra.run.smithi188.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2023-11-15T04:05:13.298 INFO:teuthology.orchestra.run.smithi188.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 361 kB/s | 157 kB 00:00 2023-11-15T04:05:13.398 INFO:teuthology.orchestra.run.smithi188.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 455 kB/s | 57 kB 00:00 2023-11-15T04:05:13.540 INFO:teuthology.orchestra.run.smithi188.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.8 MB/s | 457 kB 00:00 2023-11-15T04:05:13.574 INFO:teuthology.orchestra.run.smithi188.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 307 kB/s | 103 kB 00:00 2023-11-15T04:05:13.599 INFO:teuthology.orchestra.run.smithi188.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2023-11-15T04:05:13.691 INFO:teuthology.orchestra.run.smithi188.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 932 kB/s | 109 kB 00:00 2023-11-15T04:05:13.717 INFO:teuthology.orchestra.run.smithi188.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.6 MB/s | 641 kB 00:00 2023-11-15T04:05:13.867 INFO:teuthology.orchestra.run.smithi188.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2023-11-15T04:05:14.042 INFO:teuthology.orchestra.run.smithi188.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 528 kB/s | 185 kB 00:00 2023-11-15T04:05:14.068 INFO:teuthology.orchestra.run.smithi188.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 493 kB/s | 173 kB 00:00 2023-11-15T04:05:14.118 INFO:teuthology.orchestra.run.smithi188.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-11-15T04:05:14.143 INFO:teuthology.orchestra.run.smithi188.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-11-15T04:05:14.168 INFO:teuthology.orchestra.run.smithi188.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-11-15T04:05:14.194 INFO:teuthology.orchestra.run.smithi188.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2023-11-15T04:05:14.228 INFO:teuthology.orchestra.run.smithi188.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 697 kB/s | 59 kB 00:00 2023-11-15T04:05:14.261 INFO:teuthology.orchestra.run.smithi188.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 232 kB/s | 16 kB 00:00 2023-11-15T04:05:14.303 INFO:teuthology.orchestra.run.smithi188.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2023-11-15T04:05:14.337 INFO:teuthology.orchestra.run.smithi188.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-15T04:05:14.394 INFO:teuthology.orchestra.run.smithi188.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 182 kB/s | 24 kB 00:00 2023-11-15T04:05:14.421 INFO:teuthology.orchestra.run.smithi188.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 249 kB/s | 29 kB 00:00 2023-11-15T04:05:14.446 INFO:teuthology.orchestra.run.smithi188.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-11-15T04:05:14.488 INFO:teuthology.orchestra.run.smithi188.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.6 MB/s | 255 kB 00:00 2023-11-15T04:05:14.521 INFO:teuthology.orchestra.run.smithi188.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-11-15T04:05:14.563 INFO:teuthology.orchestra.run.smithi188.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 304 kB/s | 23 kB 00:00 2023-11-15T04:05:14.589 INFO:teuthology.orchestra.run.smithi188.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2023-11-15T04:05:14.593 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:14.593 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 31 MB 00:02 2023-11-15T04:05:14.671 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:15.028 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:15.031 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:05:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:05:16.222 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:05:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:05:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:05:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:05:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:05:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:05:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:05:16.600 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:05:16.850 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:05:17.073 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:05:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:05:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:17.112 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:05:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:05:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:05:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:17.113 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:17.229 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:05:17.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2023-11-15T04:05:17.427 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:05:17.628 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:05:17.798 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:05:18.016 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:05:18.261 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:05:18.322 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:00 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:18.503 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:05:18.740 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:05:19.094 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:05:19.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:05:19.541 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:05:19.791 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:05:20.000 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:05:20.179 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:20.182 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:05:20.183 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:20.183 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:05:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:05:20.183 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:05:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:05:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:05:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:05:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:05:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:05:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:05:20.186 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:05:20.186 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:05:20.186 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:05:20.186 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:05:20.187 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:05:20.187 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:05:20.187 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:05:20.187 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:05:20.187 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:05:20.188 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:05:20.188 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:05:20.188 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:05:20.188 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:05:20.188 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:05:20.189 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:05:20.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:20.190 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:05:20.190 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:20.190 INFO:teuthology.orchestra.run.smithi038.stdout:Install 32 Packages 2023-11-15T04:05:20.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:20.191 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 M 2023-11-15T04:05:20.191 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 111 M 2023-11-15T04:05:20.192 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:05:20.317 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:05:20.546 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:05:20.598 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 129 kB/s | 6.3 kB 00:00 2023-11-15T04:05:20.657 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-11-15T04:05:20.710 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:05:20.715 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-11-15T04:05:20.741 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T04:05:20.824 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2023-11-15T04:05:20.934 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:05:21.092 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2023-11-15T04:05:21.117 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2023-11-15T04:05:21.142 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 392 kB/s | 157 kB 00:00 2023-11-15T04:05:21.195 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:05:21.276 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 360 kB/s | 57 kB 00:00 2023-11-15T04:05:21.368 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2023-11-15T04:05:21.410 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 326 kB/s | 103 kB 00:00 2023-11-15T04:05:21.435 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.5 MB/s | 249 kB 00:00 2023-11-15T04:05:21.484 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:05:21.527 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 935 kB/s | 109 kB 00:00 2023-11-15T04:05:21.585 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.9 MB/s | 641 kB 00:00 2023-11-15T04:05:21.611 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 248 kB/s | 44 kB 00:00 2023-11-15T04:05:21.684 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:05:21.686 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2023-11-15T04:05:21.728 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 923 kB/s | 185 kB 00:00 2023-11-15T04:05:21.753 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-11-15T04:05:21.778 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-11-15T04:05:21.804 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 973 kB/s | 49 kB 00:00 2023-11-15T04:05:21.829 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-11-15T04:05:21.854 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-11-15T04:05:21.871 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 371 kB/s | 16 kB 00:00 2023-11-15T04:05:21.926 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2023-11-15T04:05:21.955 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-11-15T04:05:21.972 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-11-15T04:05:21.989 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 468 kB/s | 29 kB 00:00 2023-11-15T04:05:22.011 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:05:22.014 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 659 kB/s | 27 kB 00:00 2023-11-15T04:05:22.040 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2023-11-15T04:05:22.065 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-11-15T04:05:22.090 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2023-11-15T04:05:22.107 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 926 kB/s | 85 kB 00:00 2023-11-15T04:05:22.109 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:22.109 INFO:teuthology.orchestra.run.smithi038.stdout:Total 16 MB/s | 31 MB 00:01 2023-11-15T04:05:22.199 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:05:22.242 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:05:22.242 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:05:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:05:22.578 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:05:22.582 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:05:22.725 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:05:22.945 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:05:23.784 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:05:23.992 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:05:24.213 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:05:24.465 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:05:24.485 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:05:24.661 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:05:24.720 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:05:24.866 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:05:25.049 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:05:25.154 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:05:25.256 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:05:25.345 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:05:25.417 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:05:25.568 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:05:25.639 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:05:25.757 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:05:25.775 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:05:25.898 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:05:25.958 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:05:26.183 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:05:26.303 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:05:26.304 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:05:26.304 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:05:26.304 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:05:26.304 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:05:26.304 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:05:26.305 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:05:26.305 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:05:26.305 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:05:26.306 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:05:26.306 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:05:26.306 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:05:26.306 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:05:26.306 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:05:26.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:05:26.308 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:05:26.308 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:05:26.308 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:05:26.308 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:05:26.308 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:05:26.309 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:05:26.310 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:05:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:05:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:05:27.144 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:05:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:05:27.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:05:27.404 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:27.404 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:05:27.404 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:27.405 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:27.405 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:27.405 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:05:27.405 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:27.405 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:27.406 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:05:27.406 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:05:27.406 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:05:27.406 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:05:27.407 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:05:27.407 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:05:27.407 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:05:27.407 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:05:27.407 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:05:27.408 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:05:27.408 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:05:27.408 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:05:27.408 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:05:27.408 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:05:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:05:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:05:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:05:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:05:27.409 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:05:27.410 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:05:27.410 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:05:27.410 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:05:27.411 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:05:27.411 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:05:27.411 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:05:27.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:05:27.412 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:27.413 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:27.603 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:05:27.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2023-11-15T04:05:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:05:28.200 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:05:28.402 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:05:28.560 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:28.607 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:05:28.812 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:05:29.105 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:05:29.365 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:05:29.582 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:05:30.018 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:05:30.121 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:05:30.233 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:30.234 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:05:30.235 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:30.362 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2023-11-15T04:05:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:05:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:05:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:05:31.220 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:32.807 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:05:32.876 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:32.877 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:32.878 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:05:32.878 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:32.878 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:32.878 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:05:32.879 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:32.879 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:32.879 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:32.879 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:05:32.879 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:32.880 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 71 k 2023-11-15T04:05:32.880 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 290 k 2023-11-15T04:05:32.880 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:32.942 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.1 MB/s | 71 kB 00:00 2023-11-15T04:05:32.943 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:32.943 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.1 MB/s | 71 kB 00:00 2023-11-15T04:05:32.944 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:32.950 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:32.951 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:32.952 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:32.956 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:33.002 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:33.009 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:05:33.476 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:33.493 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:05:33.689 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:33.726 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:05:33.916 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:33.974 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:05:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:05:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:05:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:05:34.568 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:34.569 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:05:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:05:34.569 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:34.570 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:05:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:05:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:05:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:05:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:05:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:05:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:05:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:05:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:05:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:05:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:05:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:05:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:05:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:05:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:05:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:05:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:05:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:05:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:05:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:05:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:05:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:05:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:05:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:05:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:05:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:05:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:05:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:05:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:05:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:05:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:05:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:05:34.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:05:35.700 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:36.635 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:05:36.636 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:36.636 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:36.636 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:36.636 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:36.637 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:36.637 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:05:36.637 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:36.637 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:36.637 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:05:36.638 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:05:36.638 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:05:36.638 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:05:36.639 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:05:36.639 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:05:36.639 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:05:36.639 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:05:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:05:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:05:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:05:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:05:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:05:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:05:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:05:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:05:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:05:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:05:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:05:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:05:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:05:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:05:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:05:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:05:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:05:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:05:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:36.644 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:36.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2023-11-15T04:05:37.369 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:37.371 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:37.371 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2023-11-15T04:05:37.371 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:37.371 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:05:37.372 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:37.373 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 163 k 2023-11-15T04:05:37.373 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 587 k 2023-11-15T04:05:37.373 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:37.427 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.9 MB/s | 163 kB 00:00 2023-11-15T04:05:37.428 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:37.428 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.8 MB/s | 163 kB 00:00 2023-11-15T04:05:37.428 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:37.438 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:37.486 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:37.489 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:37.600 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:37.775 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:37.787 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:38.205 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:38.901 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:38.902 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:05:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:38.902 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:38.902 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:39.182 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2023-11-15T04:05:39.369 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:05:39.613 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:39.614 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:05:39.614 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:39.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2023-11-15T04:05:40.052 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:40.702 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:41.479 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:05:41.695 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:41.696 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:05:41.697 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:42.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:05:42.571 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:42.572 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:42.572 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:05:42.572 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:42.572 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:05:42.572 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:05:42.573 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:42.573 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:05:42.573 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:42.573 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:05:42.573 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:42.574 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 71 k 2023-11-15T04:05:42.574 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 290 k 2023-11-15T04:05:42.574 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:05:42.622 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-11-15T04:05:42.623 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:42.623 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.4 MB/s | 71 kB 00:00 2023-11-15T04:05:42.623 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:05:42.631 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:05:42.631 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:05:42.633 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:05:42.637 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:05:42.679 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:05:43.117 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:43.254 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:43.430 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:05:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:44.420 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:05:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:44.421 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:44.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:05:44.788 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:44.789 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:44.789 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:05:44.789 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:44.789 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:44.790 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:05:44.790 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:05:44.790 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:05:44.790 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:05:44.790 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:05:44.791 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:05:44.791 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:05:44.791 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:44.791 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:44.791 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:05:44.792 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2023-11-15T04:05:44.792 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:44.793 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.0 M 2023-11-15T04:05:44.793 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 15 M 2023-11-15T04:05:44.793 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:45.232 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 295 kB/s | 24 kB 00:00 2023-11-15T04:05:45.263 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 79 kB/s | 8.9 kB 00:00 2023-11-15T04:05:45.322 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-11-15T04:05:45.397 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 258 kB/s | 43 kB 00:00 2023-11-15T04:05:45.547 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 118 kB/s | 34 kB 00:00 2023-11-15T04:05:45.655 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:45.814 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 399 kB/s | 196 kB 00:00 2023-11-15T04:05:45.816 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:45.816 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.9 MB/s | 3.0 MB 00:01 2023-11-15T04:05:45.826 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:45.836 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:45.836 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:46.000 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:46.003 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:46.549 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:05:46.819 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:05:47.045 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:05:47.208 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:05:47.529 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:47.530 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:47.530 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2023-11-15T04:05:47.530 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:47.531 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:05:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:05:47.531 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:47.531 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:05:47.531 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:47.532 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:05:47.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:47.533 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 163 k 2023-11-15T04:05:47.533 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 587 k 2023-11-15T04:05:47.533 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:05:47.577 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.5 MB/s | 163 kB 00:00 2023-11-15T04:05:47.577 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:47.578 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.4 MB/s | 163 kB 00:00 2023-11-15T04:05:47.578 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:05:47.590 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:05:47.591 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:05:47.646 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:05:47.650 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:05:47.768 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:05:47.966 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:48.259 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:05:48.435 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:48.522 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:05:48.795 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:05:48.795 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:05:48.795 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:05:48.795 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:05:48.796 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:05:48.796 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:05:49.163 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:05:49.164 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:49.164 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:49.164 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:05:49.164 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:49.165 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:49.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2023-11-15T04:05:49.493 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:05:49.494 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:49.494 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:05:49.494 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:05:49.494 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:05:49.495 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:05:49.495 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:05:49.495 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:05:49.495 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:05:49.496 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:49.496 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:05:49.675 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:05:50.399 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:32 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:50.532 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:34 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:05:52.026 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:05:52.186 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:05:52.188 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================================ 2023-11-15T04:05:52.188 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:05:52.188 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================================ 2023-11-15T04:05:52.188 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:05:52.188 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:05:52.189 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:05:52.189 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:05:52.189 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:05:52.189 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:05:52.189 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:05:52.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:05:52.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:05:52.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:05:52.190 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:52.190 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:05:52.191 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================================ 2023-11-15T04:05:52.191 INFO:teuthology.orchestra.run.smithi188.stdout:Install 8 Packages 2023-11-15T04:05:52.191 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:05:52.192 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 37 M 2023-11-15T04:05:52.192 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 210 M 2023-11-15T04:05:52.193 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:05:52.268 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:52.269 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:05:52.270 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:52.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:05:52.411 INFO:teuthology.orchestra.run.smithi188.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 60 MB/s | 8.6 MB 00:00 2023-11-15T04:05:52.972 INFO:teuthology.orchestra.run.smithi188.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.7 MB 00:00 2023-11-15T04:05:53.076 INFO:teuthology.orchestra.run.smithi188.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.0 MB/s | 4.9 MB 00:00 2023-11-15T04:05:53.139 INFO:teuthology.orchestra.run.smithi188.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2023-11-15T04:05:53.165 INFO:teuthology.orchestra.run.smithi188.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-11-15T04:05:53.273 INFO:teuthology.orchestra.run.smithi188.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-11-15T04:05:53.356 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:05:53.383 INFO:teuthology.orchestra.run.smithi188.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.6 MB/s | 645 kB 00:00 2023-11-15T04:05:53.592 INFO:teuthology.orchestra.run.smithi188.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 27 MB/s | 14 MB 00:00 2023-11-15T04:05:53.593 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:53.593 INFO:teuthology.orchestra.run.smithi188.stdout:Total 26 MB/s | 37 MB 00:01 2023-11-15T04:05:53.819 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:05:53.838 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:05:53.838 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:05:54.169 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:05:54.172 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:05:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:05:54.983 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:05:55.103 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:05:55.225 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:05:55.225 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:55.226 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:05:55.226 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:55.226 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:05:55.226 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:05:55.226 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:05:55.227 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:05:55.227 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:05:55.227 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:05:55.227 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:05:55.227 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:05:55.228 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:55.228 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:05:55.228 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:05:55.228 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2023-11-15T04:05:55.228 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:55.230 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.0 M 2023-11-15T04:05:55.231 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 15 M 2023-11-15T04:05:55.231 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:05:55.258 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:05:55.363 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 472 kB/s | 24 kB 00:00 2023-11-15T04:05:55.380 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 131 kB/s | 8.9 kB 00:00 2023-11-15T04:05:55.447 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-11-15T04:05:55.472 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 390 kB/s | 43 kB 00:00 2023-11-15T04:05:55.489 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 310 kB/s | 34 kB 00:00 2023-11-15T04:05:55.573 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 MB/s | 196 kB 00:00 2023-11-15T04:05:55.576 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:05:55.576 INFO:teuthology.orchestra.run.smithi038.stdout:Total 8.4 MB/s | 3.0 MB 00:00 2023-11-15T04:05:55.589 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:05:55.601 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:05:55.601 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:05:55.768 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:05:55.771 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:05:55.839 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:05:56.058 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:05:56.070 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:05:56.276 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:05:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:05:56.674 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:05:56.784 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:05:56.858 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:05:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:05:57.542 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:05:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:05:58.364 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:05:58.676 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:05:58.676 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:05:58.676 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:05:58.676 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:05:58.677 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:05:58.677 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:05:59.101 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:05:59.365 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:05:59.441 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:05:59.441 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:59.441 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:05:59.441 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:05:59.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:05:59.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:05:59.442 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:05:59.442 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:05:59.442 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:05:59.443 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:05:59.443 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:05:59.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:06:00.561 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:01.135 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:06:02.430 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:02.431 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================================ 2023-11-15T04:06:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:06:02.431 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================================ 2023-11-15T04:06:02.432 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:06:02.432 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:06:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:06:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:06:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:06:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:06:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:06:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:06:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:06:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:02.434 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:02.434 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================================ 2023-11-15T04:06:02.434 INFO:teuthology.orchestra.run.smithi038.stdout:Install 8 Packages 2023-11-15T04:06:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:02.436 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:06:02.437 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 37 M 2023-11-15T04:06:02.438 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 210 M 2023-11-15T04:06:02.438 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:02.639 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:06:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:06:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:06:03.000 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:06:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:06:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:06:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:06:03.001 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:06:03.002 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:06:03.453 INFO:teuthology.orchestra.run.smithi038.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 11 MB/s | 8.6 MB 00:00 2023-11-15T04:06:03.737 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:06:03.738 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:03.738 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:03.738 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:06:03.738 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:06:03.738 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:06:03.739 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:06:03.739 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:06:03.739 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:06:03.739 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:06:03.739 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:06:03.740 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:03.740 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:03.846 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:06:04.130 INFO:teuthology.orchestra.run.smithi038.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.3 MB/s | 4.7 MB 00:01 2023-11-15T04:06:04.239 INFO:teuthology.orchestra.run.smithi038.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.8 MB/s | 3.7 MB 00:00 2023-11-15T04:06:04.264 INFO:teuthology.orchestra.run.smithi038.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-11-15T04:06:04.331 INFO:teuthology.orchestra.run.smithi038.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.9 MB 00:01 2023-11-15T04:06:04.440 INFO:teuthology.orchestra.run.smithi038.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-11-15T04:06:04.465 INFO:teuthology.orchestra.run.smithi038.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2023-11-15T04:06:04.708 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:04.782 INFO:teuthology.orchestra.run.smithi038.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 25 MB/s | 14 MB 00:00 2023-11-15T04:06:04.782 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:04.783 INFO:teuthology.orchestra.run.smithi038.stdout:Total 16 MB/s | 37 MB 00:02 2023-11-15T04:06:05.033 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:05.054 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:05.054 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:05.421 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:06.100 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:06.239 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:06:06.346 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:06.348 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:06:06.348 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:06.348 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:06.348 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:06:06.349 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:06:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:06:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:06:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:06:06.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:06:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:06:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:06:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:06:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:06:06.350 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:06:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:06:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:06:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:06:06.351 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:06.352 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:06.352 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:06.352 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2023-11-15T04:06:06.352 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:06.353 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.9 M 2023-11-15T04:06:06.353 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 24 M 2023-11-15T04:06:06.353 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:06.416 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:06:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:06:06.665 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 605 kB/s | 27 kB 00:00 2023-11-15T04:06:06.798 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 112 kB/s | 20 kB 00:00 2023-11-15T04:06:06.815 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 136 kB/s | 27 kB 00:00 2023-11-15T04:06:06.924 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-11-15T04:06:06.958 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-11-15T04:06:06.991 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 387 kB/s | 126 kB 00:00 2023-11-15T04:06:07.037 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 461 kB/s | 110 kB 00:00 2023-11-15T04:06:07.134 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-11-15T04:06:07.151 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 102 kB/s | 16 kB 00:00 2023-11-15T04:06:07.176 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 631 kB/s | 81 kB 00:00 2023-11-15T04:06:07.201 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-11-15T04:06:07.204 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:06:07.235 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-11-15T04:06:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:06:07.609 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.5 MB/s | 1.2 MB 00:00 2023-11-15T04:06:07.611 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:07.612 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-11-15T04:06:07.668 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:07.683 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:07.683 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:07.769 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:07.772 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:08.066 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:06:08.213 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:06:08.278 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:08.516 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:06:08.719 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:06:08.974 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:06:08.987 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:06:09.212 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:06:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:06:09.463 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:06:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:06:09.720 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:06:10.029 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:06:10.302 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:06:11.305 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:06:11.568 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:06:11.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:06:11.819 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:06:12.021 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:06:12.229 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:06:12.515 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:06:12.515 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:06:12.516 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:06:12.516 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:06:12.516 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:06:12.516 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:06:12.517 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:06:12.517 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:06:12.517 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:06:12.517 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:06:12.517 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:06:12.518 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:06:12.518 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:06:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:06:13.172 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:06:13.245 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:06:13.245 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:13.246 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:13.246 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:06:13.246 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:06:13.246 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:06:13.246 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:06:13.247 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:06:13.249 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:06:13.249 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:06:13.250 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:06:13.250 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:06:13.250 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:06:13.250 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:06:13.250 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:06:13.251 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:06:13.251 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:13.251 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:13.376 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:06:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:06:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:06:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:06:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:06:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:06:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:06:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:06:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:06:14.230 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:14.298 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:06:14.298 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:14.299 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:14.299 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:06:14.299 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:06:14.300 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:06:14.300 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:06:14.300 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:06:14.300 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:06:14.300 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:06:14.301 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:06:14.301 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:14.301 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:14.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:06:15.436 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:15.857 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:15.858 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:15.858 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-11-15T04:06:15.858 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:15.859 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:15.859 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:06:15.859 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:06:15.859 INFO:teuthology.orchestra.run.smithi188.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:06:15.859 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:06:15.860 INFO:teuthology.orchestra.run.smithi188.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:06:15.860 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:06:15.860 INFO:teuthology.orchestra.run.smithi188.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:06:15.860 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:15.860 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:15.861 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:15.861 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2023-11-15T04:06:15.861 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:15.862 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 5.8 M 2023-11-15T04:06:15.862 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 24 M 2023-11-15T04:06:15.862 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:16.275 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2023-11-15T04:06:16.617 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 62 kB/s | 21 kB 00:00 2023-11-15T04:06:16.742 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2023-11-15T04:06:16.868 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2023-11-15T04:06:17.001 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.2 MB/s | 4.8 MB 00:00 2023-11-15T04:06:17.027 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 172 kB/s | 49 kB 00:00 2023-11-15T04:06:17.029 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:17.029 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.0 MB/s | 5.8 MB 00:01 2023-11-15T04:06:17.081 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:17.091 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:17.091 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:17.199 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:17.203 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:17.314 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:17.316 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:17.316 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:06:17.316 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:17.317 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:17.317 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:06:17.317 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:06:17.317 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:06:17.317 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:06:17.318 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:06:17.318 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:06:17.318 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:06:17.318 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:06:17.319 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:06:17.319 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:06:17.319 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:06:17.319 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:06:17.319 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:06:17.320 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:06:17.320 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:17.320 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:17.320 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:17.321 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2023-11-15T04:06:17.321 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:17.322 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.9 M 2023-11-15T04:06:17.322 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2023-11-15T04:06:17.322 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:17.501 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:17.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:06:17.996 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 525 kB/s | 27 kB 00:00 2023-11-15T04:06:18.081 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:06:18.130 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 108 kB/s | 20 kB 00:00 2023-11-15T04:06:18.155 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 126 kB/s | 27 kB 00:00 2023-11-15T04:06:18.264 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-11-15T04:06:18.330 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 653 kB/s | 43 kB 00:00 2023-11-15T04:06:18.364 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 470 kB/s | 110 kB 00:00 2023-11-15T04:06:18.390 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 322 kB/s | 126 kB 00:00 2023-11-15T04:06:18.498 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 196 kB/s | 33 kB 00:00 2023-11-15T04:06:18.524 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 103 kB/s | 16 kB 00:00 2023-11-15T04:06:18.549 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 550 kB/s | 81 kB 00:00 2023-11-15T04:06:18.608 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 769 kB/s | 64 kB 00:00 2023-11-15T04:06:18.633 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 735 kB/s | 61 kB 00:00 2023-11-15T04:06:18.808 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2023-11-15T04:06:18.811 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:18.811 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-11-15T04:06:18.874 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:18.892 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:18.892 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:18.991 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:18.994 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:19.004 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:06:19.380 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:06:19.587 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:19.651 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:06:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:06:20.069 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:06:20.070 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:06:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:06:20.366 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:06:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:06:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:06:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:06:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:06:20.367 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:06:20.579 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:06:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:06:21.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:06:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:21.056 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:06:21.056 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:06:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:06:21.057 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:21.057 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:21.057 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:06:21.242 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2023-11-15T04:06:21.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:06:21.654 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:06:22.105 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:22.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:06:23.095 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:06:23.304 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:06:23.520 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:06:23.695 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:06:23.772 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:23.773 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:23.773 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:23.773 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:23.773 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:23.774 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:06:23.774 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-11-15T04:06:23.774 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:06:23.774 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:23.774 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:23.775 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:23.775 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2023-11-15T04:06:23.775 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:23.776 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 968 k 2023-11-15T04:06:23.776 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 3.2 M 2023-11-15T04:06:23.776 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:24.033 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:06:24.033 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:06:24.033 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:06:24.033 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:06:24.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:06:24.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:06:24.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:06:24.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:06:24.034 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:06:24.035 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:06:24.035 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:06:24.035 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:06:24.035 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:06:24.055 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 9.5 MB/s | 885 kB 00:00 2023-11-15T04:06:24.138 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 477 kB/s | 83 kB 00:00 2023-11-15T04:06:24.139 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:24.139 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.6 MB/s | 968 kB 00:00 2023-11-15T04:06:24.149 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:24.169 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:24.169 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:24.254 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:24.257 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:24.445 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:24.743 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:06:24.886 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:06:24.886 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:24.886 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:24.886 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:06:24.887 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:06:24.887 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:06:24.887 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:06:24.887 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:06:24.887 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:06:24.888 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:06:24.888 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:06:24.888 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:06:24.888 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:06:24.888 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:06:24.889 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:06:24.889 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:06:24.889 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:24.889 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:24.959 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:06:25.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:06:25.354 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:06:25.354 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:06:25.999 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:26.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:06:26.060 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:26.060 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:26.060 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:06:26.061 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:26.061 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:26.284 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2023-11-15T04:06:27.159 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:27.858 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:27.859 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-15T04:06:27.860 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:27.860 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:06:27.861 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:06:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:06:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:06:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:06:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:06:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:06:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:27.862 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:27.862 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:27.863 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2023-11-15T04:06:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:27.863 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 5.8 M 2023-11-15T04:06:27.864 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 24 M 2023-11-15T04:06:27.864 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:28.015 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2023-11-15T04:06:28.191 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 121 kB/s | 21 kB 00:00 2023-11-15T04:06:28.349 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-11-15T04:06:28.374 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2023-11-15T04:06:28.549 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 244 kB/s | 49 kB 00:00 2023-11-15T04:06:28.651 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.0 MB/s | 4.8 MB 00:00 2023-11-15T04:06:28.653 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:28.654 INFO:teuthology.orchestra.run.smithi038.stdout:Total 7.3 MB/s | 5.8 MB 00:00 2023-11-15T04:06:28.711 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:28.723 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:28.723 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:28.830 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:28.830 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:28.831 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:28.831 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:06:28.831 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:28.832 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:28.832 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:28.832 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:06:28.832 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:28.833 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:28.833 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 131 k 2023-11-15T04:06:28.834 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 439 k 2023-11-15T04:06:28.834 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:28.836 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:28.884 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2023-11-15T04:06:28.885 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:28.885 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.4 MB/s | 131 kB 00:00 2023-11-15T04:06:28.885 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:28.890 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:28.890 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:28.941 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:28.945 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:29.056 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:29.158 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:29.291 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:29.561 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:29.681 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:06:30.005 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:06:30.259 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:30.260 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:30.260 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:30.260 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:06:30.260 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:30.261 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:31.044 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:06:31.273 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2023-11-15T04:06:31.432 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:06:31.678 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:06:31.869 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:06:32.189 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:06:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:06:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:06:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:06:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:06:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:06:32.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:06:32.972 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:32.972 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:32.972 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:06:32.972 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:06:32.972 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:06:32.973 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:32.973 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:33.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2023-11-15T04:06:33.670 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:33.886 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:33.887 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:33.887 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:34.005 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2023-11-15T04:06:34.130 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:34.874 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:36.005 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:36.006 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:36.006 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:36.006 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:36.007 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:36.007 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:06:36.007 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-15T04:06:36.007 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:06:36.007 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:36.008 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:36.008 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:36.008 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2023-11-15T04:06:36.008 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:36.009 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 968 k 2023-11-15T04:06:36.009 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 3.2 M 2023-11-15T04:06:36.009 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:36.270 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2023-11-15T04:06:36.345 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 591 kB/s | 83 kB 00:00 2023-11-15T04:06:36.345 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:36.346 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.8 MB/s | 968 kB 00:00 2023-11-15T04:06:36.358 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:36.383 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:36.383 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:36.486 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:36.490 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:36.536 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:36.537 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:36.537 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:36.537 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:36.537 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:06:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:36.538 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:36.538 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:36.538 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:06:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:36.539 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 24 k 2023-11-15T04:06:36.539 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 94 k 2023-11-15T04:06:36.540 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:36.600 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 392 kB/s | 24 kB 00:00 2023-11-15T04:06:36.601 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:36.601 INFO:teuthology.orchestra.run.smithi188.stdout:Total 385 kB/s | 24 kB 00:00 2023-11-15T04:06:36.601 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:36.604 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:36.604 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:36.606 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:36.609 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:36.662 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:36.847 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:06:37.128 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:06:37.131 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:06:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:06:37.823 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:37.823 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:37.824 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:06:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:37.825 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:37.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2023-11-15T04:06:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:06:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:38.338 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:06:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:38.338 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:38.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2023-11-15T04:06:38.831 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:39.601 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:21 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:40.268 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:40.486 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:40.487 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:40.488 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:40.604 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2023-11-15T04:06:41.458 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:41.459 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:41.459 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:41.460 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:06:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:41.460 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:41.460 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:41.461 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:06:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:41.461 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 131 k 2023-11-15T04:06:41.462 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 439 k 2023-11-15T04:06:41.462 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:41.470 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:41.518 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 131 kB 00:00 2023-11-15T04:06:41.519 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:41.519 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.2 MB/s | 131 kB 00:00 2023-11-15T04:06:41.519 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:41.525 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:41.526 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:41.584 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:41.587 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:41.751 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:41.976 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:42.325 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:42.907 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:43.089 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:43.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:43.089 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:43.089 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:06:43.090 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:43.090 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:43.122 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:43.124 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:43.124 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:43.237 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2023-11-15T04:06:43.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2023-11-15T04:06:44.101 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:44.315 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:26 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:45.533 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:45.749 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:45.751 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:45.751 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:45.903 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2023-11-15T04:06:45.945 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:46.187 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:46.188 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:06:46.189 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:46.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2023-11-15T04:06:46.780 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:47.584 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:48.231 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:48.446 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:48.447 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:48.448 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:48.576 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2023-11-15T04:06:49.442 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:49.454 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:49.455 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:49.456 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:49.456 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:06:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:06:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:49.457 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:06:49.457 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:06:49.457 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:06:49.457 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:49.457 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 24 k 2023-11-15T04:06:49.458 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 94 k 2023-11-15T04:06:49.458 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:06:49.522 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 371 kB/s | 24 kB 00:00 2023-11-15T04:06:49.523 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:49.523 INFO:teuthology.orchestra.run.smithi038.stdout:Total 362 kB/s | 24 kB 00:00 2023-11-15T04:06:49.523 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:06:49.527 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:06:49.528 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:06:49.530 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:06:49.534 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:06:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:06:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:50.254 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:50.886 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:50.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:50.970 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:50.970 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:06:50.970 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:06:50.970 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:06:50.970 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:51.099 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:51.100 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:51.100 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:51.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2023-11-15T04:06:51.207 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2023-11-15T04:06:52.074 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:52.124 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:34 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:53.510 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:53.723 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:53.725 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2023-11-15T04:06:53.725 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:53.730 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:53.842 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2023-11-15T04:06:53.972 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:53.973 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:06:53.974 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:54.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2023-11-15T04:06:54.714 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:38 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:55.085 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:37 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:56.370 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:06:56.371 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:56.371 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:06:56.371 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:06:56.372 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:06:56.373 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:56.373 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 89 k 2023-11-15T04:06:56.373 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 298 k 2023-11-15T04:06:56.373 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:06:56.430 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 89 kB 00:00 2023-11-15T04:06:56.431 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:06:56.432 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.5 MB/s | 89 kB 00:00 2023-11-15T04:06:56.432 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:06:56.441 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:06:56.441 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:06:56.491 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:06:56.494 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:06:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:06:56.709 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:56.859 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:56.950 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:56.952 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:06:56.952 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:06:57.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2023-11-15T04:06:57.154 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:06:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:57.821 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:06:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:06:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:06:57.822 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:06:58.045 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:40 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:06:58.051 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2023-11-15T04:06:58.916 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:06:59.648 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:06:59.895 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:06:59.896 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:06:59.897 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:00.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2023-11-15T04:07:00.584 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:07:00.584 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:00.584 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:07:00.585 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:00.585 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:07:00.585 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:07:00.585 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:00.585 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:07:00.586 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:00.586 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:07:00.586 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:00.587 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.7 M 2023-11-15T04:07:00.587 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 18 M 2023-11-15T04:07:00.588 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:07:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2023-11-15T04:07:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:07:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:Total 41 MB/s | 3.7 MB 00:00 2023-11-15T04:07:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:07:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:07:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:07:00.751 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:07:00.754 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:07:00.996 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:01.218 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:07:01.435 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:01.857 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:02.526 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:02.527 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:02.527 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:07:02.527 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:07:02.527 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:02.527 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:07:02.607 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:07:02.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2023-11-15T04:07:02.849 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:02.850 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:07:02.851 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:02.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2023-11-15T04:07:03.654 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:47 ago on Wed 15 Nov 2023 04:04:16 AM UTC. 2023-11-15T04:07:03.960 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:45 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:05.324 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-11-15T04:07:05.324 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:05.324 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-11-15T04:07:05.325 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:05.325 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-11-15T04:07:05.325 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:07:05.325 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:05.325 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-11-15T04:07:05.326 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-11-15T04:07:05.326 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2023-11-15T04:07:05.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:05.326 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 173 k 2023-11-15T04:07:05.326 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 649 k 2023-11-15T04:07:05.327 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-11-15T04:07:05.420 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 173 kB 00:00 2023-11-15T04:07:05.421 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:07:05.421 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.8 MB/s | 173 kB 00:00 2023-11-15T04:07:05.421 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-11-15T04:07:05.431 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-11-15T04:07:05.431 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-11-15T04:07:05.481 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-11-15T04:07:05.484 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-11-15T04:07:05.570 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:07:05.610 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-11-15T04:07:05.816 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:05.816 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:05.817 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:07:05.817 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:05.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2023-11-15T04:07:06.107 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-11-15T04:07:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:07:06.810 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:07:06.810 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-11-15T04:07:06.940 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:48 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:07.035 DEBUG:teuthology.parallel:result is None 2023-11-15T04:07:08.549 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:07:08.789 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:08.791 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-15T04:07:08.791 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:08.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2023-11-15T04:07:09.898 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:51 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:11.745 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:11.746 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:11.746 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:07:11.746 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:11.746 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:07:11.747 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:11.748 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 89 k 2023-11-15T04:07:11.748 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 298 k 2023-11-15T04:07:11.749 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:07:11.795 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-11-15T04:07:11.796 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:07:11.796 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.8 MB/s | 89 kB 00:00 2023-11-15T04:07:11.796 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:07:11.808 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:07:11.808 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:07:11.864 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:07:11.868 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:07:11.987 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:07:12.163 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:12.510 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:13.281 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:13.281 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:13.282 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:07:13.282 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:07:13.282 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:13.282 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:13.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2023-11-15T04:07:14.512 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:56 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:16.361 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:16.362 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:07:16.363 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:16.363 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:07:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:07:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.7 M 2023-11-15T04:07:16.364 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 18 M 2023-11-15T04:07:16.365 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:07:16.480 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-11-15T04:07:16.480 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:07:16.481 INFO:teuthology.orchestra.run.smithi038.stdout:Total 31 MB/s | 3.7 MB 00:00 2023-11-15T04:07:16.481 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:07:16.495 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:07:16.495 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:07:16.566 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:07:16.569 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:07:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:07:17.371 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:18.560 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:18.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:18.560 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:07:18.561 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:07:18.561 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:18.561 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:18.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2023-11-15T04:07:19.787 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:01 ago on Wed 15 Nov 2023 04:04:18 AM UTC. 2023-11-15T04:07:21.647 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-15T04:07:21.648 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:21.648 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-15T04:07:21.648 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:21.649 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-15T04:07:21.649 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:07:21.649 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:21.649 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-15T04:07:21.649 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-15T04:07:21.650 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-15T04:07:21.650 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:21.650 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 173 k 2023-11-15T04:07:21.651 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 649 k 2023-11-15T04:07:21.651 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-15T04:07:21.701 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2023-11-15T04:07:21.701 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:07:21.702 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.3 MB/s | 173 kB 00:00 2023-11-15T04:07:21.702 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-15T04:07:21.714 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-15T04:07:21.715 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-15T04:07:21.773 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-15T04:07:21.777 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-15T04:07:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-15T04:07:22.137 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:23.310 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:07:23.310 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:23.311 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-15T04:07:23.311 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:07:23.311 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-15T04:07:23.311 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-15T04:07:23.544 DEBUG:teuthology.parallel:result is None 2023-11-15T04:07:23.545 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:07:23.546 INFO:teuthology.packaging:ref: None 2023-11-15T04:07:23.546 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:07:23.546 INFO:teuthology.packaging:branch: None 2023-11-15T04:07:23.547 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:07:23.745 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:07:23.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:07:23.893 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:07:23.922 INFO:teuthology.orchestra.run.smithi038.stdout:16.2.4-0.el8 2023-11-15T04:07:23.923 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:07:23.924 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:07:23.924 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:07:23.924 INFO:teuthology.packaging:ref: None 2023-11-15T04:07:23.925 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:07:23.925 INFO:teuthology.packaging:branch: None 2023-11-15T04:07:23.925 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:07:24.134 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:07:24.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:07:24.355 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:07:24.382 INFO:teuthology.orchestra.run.smithi188.stdout:16.2.4-0.el8 2023-11-15T04:07:24.383 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:07:24.383 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:07:24.384 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T04:07:24.384 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:07:24.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:07:24.422 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:07:24.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:07:24.456 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T04:07:24.456 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:07:24.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:07:24.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:07:24.571 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:07:24.572 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:07:24.603 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:07:24.673 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T04:07:24.674 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:07:24.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:07:24.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:07:24.784 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:07:24.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:07:24.815 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:07:24.885 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T04:07:24.886 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:07:24.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:07:24.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:07:24.998 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:07:24.998 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:07:25.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:07:25.097 INFO:teuthology.run_tasks:Running task print... 2023-11-15T04:07:25.107 INFO:teuthology.task.print:**** done install task... 2023-11-15T04:07:25.107 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T04:07:25.218 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:07:25.218 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T04:07:25.219 INFO:tasks.cephadm:Cluster fsid is 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:25.219 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T04:07:25.220 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T04:07:25.220 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi038': '172.21.15.38', 'mon.smithi188': '172.21.15.188'} 2023-11-15T04:07:25.220 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T04:07:25.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2023-11-15T04:07:25.256 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2023-11-15T04:07:25.288 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T04:07:25.289 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:07 /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.435 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.579 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:07 /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.579 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.604 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:07:25.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2023-11-15T04:07:25.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2023-11-15T04:07:25.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2023-11-15T04:07:25.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2023-11-15T04:07:25.781 INFO:tasks.cephadm:Writing seed config... 2023-11-15T04:07:25.782 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T04:07:25.783 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T04:07:25.783 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T04:07:25.783 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T04:07:25.784 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T04:07:25.784 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T04:07:25.784 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T04:07:25.784 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T04:07:25.785 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T04:07:25.785 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T04:07:25.785 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T04:07:25.785 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T04:07:25.786 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T04:07:25.786 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T04:07:25.786 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T04:07:25.786 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T04:07:25.787 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T04:07:25.787 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T04:07:25.787 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T04:07:25.788 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T04:07:25.788 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T04:07:25.788 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T04:07:25.789 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T04:07:25.789 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T04:07:25.790 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T04:07:25.790 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T04:07:25.791 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T04:07:25.791 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T04:07:25.792 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T04:07:25.792 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T04:07:25.792 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T04:07:25.792 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T04:07:25.793 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T04:07:25.793 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T04:07:25.793 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T04:07:25.794 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T04:07:25.794 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T04:07:25.794 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T04:07:25.795 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:07:25.795 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T04:07:25.818 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T04:07:25.819 DEBUG:teuthology.orchestra.run.smithi038:mon.smithi038> sudo journalctl -f -n 0 -u ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service 2023-11-15T04:07:25.861 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T04:07:25.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.38 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:07:25.894 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:-- Logs begin at Wed 2023-11-15 03:54:48 UTC. -- 2023-11-15T04:07:26.193 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:07:27.062 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.6.1 2023-11-15T04:07:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying podman|docker is present... 2023-11-15T04:07:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:07:27.099 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.6.1 2023-11-15T04:07:27.111 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying lvm2 is present... 2023-11-15T04:07:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying time synchronization is in place... 2023-11-15T04:07:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:07:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:07:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:07:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:07:27.129 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:07:27.134 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: enabled 2023-11-15T04:07:27.135 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:07:27.141 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: active 2023-11-15T04:07:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:07:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:Repeating the final host check... 2023-11-15T04:07:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:07:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 4.6.1 2023-11-15T04:07:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:podman|docker (/bin/podman) is present 2023-11-15T04:07:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl is present 2023-11-15T04:07:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:lvcreate is present 2023-11-15T04:07:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:07:27.188 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:07:27.188 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:07:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:07:27.195 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:07:27.201 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: enabled 2023-11-15T04:07:27.202 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:07:27.208 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: active 2023-11-15T04:07:27.209 INFO:teuthology.orchestra.run.smithi038.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:07:27.209 INFO:teuthology.orchestra.run.smithi038.stderr:Host looks OK 2023-11-15T04:07:27.209 INFO:teuthology.orchestra.run.smithi038.stderr:Cluster fsid: 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:27.210 INFO:teuthology.orchestra.run.smithi038.stderr:Acquiring lock 140269897492968 on /run/cephadm/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:07:27.210 INFO:teuthology.orchestra.run.smithi038.stderr:Lock 140269897492968 acquired on /run/cephadm/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:07:27.210 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying IP 172.21.15.38 port 3300 ... 2023-11-15T04:07:27.211 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying IP 172.21.15.38 port 6789 ... 2023-11-15T04:07:27.211 INFO:teuthology.orchestra.run.smithi038.stderr:Base mon IP is 172.21.15.38, final addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2023-11-15T04:07:27.212 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /sbin/ip route ls 2023-11-15T04:07:27.213 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2023-11-15T04:07:27.213 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2023-11-15T04:07:27.214 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T04:07:27.216 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T04:07:27.218 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-15T04:07:27.218 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2023-11-15T04:07:27.218 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T04:07:27.220 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T04:07:27.220 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T04:07:27.220 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:07:27.221 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-15T04:07:27.221 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15fc/64 scope link 2023-11-15T04:07:27.221 INFO:teuthology.orchestra.run.smithi038.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:07:27.222 INFO:teuthology.orchestra.run.smithi038.stderr:Mon IP 172.21.15.38 is in CIDR network 172.21.0.0/20 2023-11-15T04:07:27.222 INFO:teuthology.orchestra.run.smithi038.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T04:07:27.223 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:07:27.223 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T04:07:27.324 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:07:33.784 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Getting image source signatures 2023-11-15T04:07:33.793 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T04:07:33.793 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T04:07:46.472 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:07:46.530 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T04:07:46.845 INFO:teuthology.orchestra.run.smithi038.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:07:46.858 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T04:07:47.495 INFO:teuthology.orchestra.run.smithi038.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:07:47.837 INFO:teuthology.orchestra.run.smithi038.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:07:47.838 INFO:teuthology.orchestra.run.smithi038.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T04:07:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T04:07:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:stat: 167 167 2023-11-15T04:07:48.484 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial keys... 2023-11-15T04:07:48.485 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:07:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAURFRla20BLRAALmvqGrbCSLwoykZh8cHpGQ== 2023-11-15T04:07:49.138 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:07:49.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAVRFRlJZUcGRAAK1BWrxAroJnLjXTvFu2kvQ== 2023-11-15T04:07:49.808 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:07:50.108 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-authtool: AQAWRFRlhm70BRAAjhZ1n1aEV9yH3Iv2rH+4lw== 2023-11-15T04:07:50.475 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial monmap... 2023-11-15T04:07:50.476 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmppoczlcld:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 --addv smithi038 [v2:172.21.15.38:3300,v1:172.21.15.38:6789] /tmp/monmap 2023-11-15T04:07:50.800 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:07:50.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:50.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:07:51.194 INFO:teuthology.orchestra.run.smithi038.stderr:monmaptool for smithi038 [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:07:51.195 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: set fsid to 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:51.195 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:07:51.195 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:07:51.195 INFO:teuthology.orchestra.run.smithi038.stderr:Creating mon... 2023-11-15T04:07:51.196 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/mon.smithi038:/var/lib/ceph/mon/ceph-smithi038:z -v /tmp/ceph-tmpyznxmoil:/tmp/keyring:z -v /tmp/ceph-tmppoczlcld:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi038 --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T04:07:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.526+0000 7f036ec00700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:07:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.526+0000 7f036ec00700 1 imported monmap: 2023-11-15T04:07:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T04:07:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T04:07:50.797737+0000 2023-11-15T04:07:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: created 2023-11-15T04:07:50.797737+0000 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.smithi038 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.526+0000 7f036ec00700 0 /usr/bin/ceph-mon: set fsid to 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:07:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:07:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:07:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: DB SUMMARY 2023-11-15T04:07:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi038/store.db dir, Total Num: 0, files: 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi038/store.db: 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:07:51.564 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T04:07:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:07:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.env: 0x559d76cb21c0 2023-11-15T04:07:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:07:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.info_log: 0x559d78552ec0 2023-11-15T04:07:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:07:51.566 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:07:51.566 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:07:51.566 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:07:51.566 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:07:51.566 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:07:51.567 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:07:51.568 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:07:51.568 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:07:51.568 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:07:51.568 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi038/store.db 2023-11-15T04:07:51.568 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:07:51.569 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:07:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:07:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:07:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:07:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.write_buffer_manager: 0x559d78559530 2023-11-15T04:07:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:07:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:07:51.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:07:51.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:07:51.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:07:51.572 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:07:51.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:07:51.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:07:51.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:07:51.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.row_cache: None 2023-11-15T04:07:51.573 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:07:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:07:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:07:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:07:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:07:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:07:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:07:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:07:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:07:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:07:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:07:51.576 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:07:51.577 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:07:51.577 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:07:51.577 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:07:51.577 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:07:51.577 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:07:51.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:07:51.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:07:51.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:07:51.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:07:51.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:07:51.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.528+0000 7f036ec00700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.529+0000 7f036ec00700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T04:07:51.580 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000001 2023-11-15T04:07:51.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:07:51.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:07:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.merge_operator: 2023-11-15T04:07:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:07:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:07:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:07:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.531+0000 7f036ec00700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559d7846a120) 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T04:07:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T04:07:51.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T04:07:51.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T04:07:51.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:07:51.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T04:07:51.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache: 0x559d784a0d10 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T04:07:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T04:07:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T04:07:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T04:07:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T04:07:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T04:07:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T04:07:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T04:07:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T04:07:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T04:07:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T04:07:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T04:07:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T04:07:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:07:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T04:07:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T04:07:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:07:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:07:51.590 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:07:51.590 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:07:51.590 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:07:51.590 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:07:51.591 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:07:51.592 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:07:51.593 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:07:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:07:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:07:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:07:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:07:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:07:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:07:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:07:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:07:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:07:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:07:51.597 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:07:51.597 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:07:51.597 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:07:51.597 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:07:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:07:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:07:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:07:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:07:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:07:51.599 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:07:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:07:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:07:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:07:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:07:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:07:51.601 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:07:51.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:07:51.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:07:51.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:07:51.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:07:51.602 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:07:51.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:07:51.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:07:51.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:07:51.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.532+0000 7f036ec00700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:07:51.603 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.533+0000 7f036ec00700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:07:51.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.533+0000 7f036ec00700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T04:07:51.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.604 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.534+0000 7f036ec00700 4 rocksdb: DB pointer 0x559d78565800 2023-11-15T04:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.534+0000 7f0357a99700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.534+0000 7f0357a99700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T04:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:07:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:07:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:07:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:07:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:07:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:07:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:07:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:07:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:07:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:07:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:07:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:07:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:07:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.611 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:07:51.611 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.611 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:07:51.611 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:51.611 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:07:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:07:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:07:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:07:51.614 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:07:51.614 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:07:51.614 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:51.614 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:07:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:07:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: 2023-11-15T04:07:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.537+0000 7f036ec00700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T04:07:51.616 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.537+0000 7f036ec00700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T04:07:51.616 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:07:51.537+0000 7f036ec00700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi038 for mon.smithi038 2023-11-15T04:07:51.923 INFO:teuthology.orchestra.run.smithi038.stderr:create mon.smithi038 on 2023-11-15T04:07:51.924 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:51.930 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl disable ceph.target 2023-11-15T04:07:51.942 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T04:07:52.037 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl enable ceph.target 2023-11-15T04:07:52.044 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T04:07:52.132 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl start ceph.target 2023-11-15T04:07:52.140 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl enable ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:07:52.147 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:07:52.147 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:07:52.235 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:07:52.243 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl daemon-reload 2023-11-15T04:07:52.341 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:07:52.370 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl reset-failed ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:07:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to reset failed state of unit ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service: Unit ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service not loaded. 2023-11-15T04:07:52.377 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl enable ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:07:52.384 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target.wants/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service → /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:07:52.479 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:07:52.801 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:52 smithi038 systemd[1]: Starting Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7... 2023-11-15T04:07:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:07:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2023-11-15T04:07:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:07:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:07:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2023-11-15T04:07:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:07:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:07:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mon to start... 2023-11-15T04:07:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mon... 2023-11-15T04:07:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/mon.smithi038:/var/lib/ceph/mon/ceph-smithi038:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T04:07:53.339 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:52 smithi038 bash[111940]: 328e6ff6a992e1187d48038884f3dfaee55dade2d83e83ba6434ab0c6d6d68e8 2023-11-15T04:07:53.340 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:52 smithi038 systemd[1]: Started Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7. 2023-11-15T04:07:53.341 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.840316+0000 mon.smithi038 (mon.0) 0 : [INF] mkfs 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:53.341 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.822810+0000 mon.smithi038 (mon.0) 1 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2023-11-15T04:07:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.248+0000 7f59b3fff700 1 Processor -- start 2023-11-15T04:07:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.248+0000 7f59b3fff700 1 -- start start 2023-11-15T04:07:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.248+0000 7f59b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.248+0000 7f59b3fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40fec40 con 0x7f59b40704a0 2023-11-15T04:07:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.249+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.249+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60686/0 (socket says 172.21.15.38:60686) 2023-11-15T04:07:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.249+0000 7f59b2ffd700 1 -- 172.21.15.38:0/2715597070 learned_addr learned my addr 172.21.15.38:0/2715597070 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.249+0000 7f59b2ffd700 1 -- 172.21.15.38:0/2715597070 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40fed80 con 0x7f59b40704a0 2023-11-15T04:07:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b2ffd700 1 --2- 172.21.15.38:0/2715597070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f59a400b5f0 tx=0x7f59a4006ee0).ready entity=mon.0 client_cookie=1a79f2790e545ec9 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b1ffb700 1 -- 172.21.15.38:0/2715597070 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a400d600 con 0x7f59b40704a0 2023-11-15T04:07:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b1ffb700 1 -- 172.21.15.38:0/2715597070 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a400d760 con 0x7f59b40704a0 2023-11-15T04:07:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b1ffb700 1 -- 172.21.15.38:0/2715597070 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a400da30 con 0x7f59b40704a0 2023-11-15T04:07:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 -- 172.21.15.38:0/2715597070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 msgr2=0x7f59b40fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 --2- 172.21.15.38:0/2715597070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f59a400b5f0 tx=0x7f59a4006ee0).stop 2023-11-15T04:07:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 -- 172.21.15.38:0/2715597070 shutdown_connections 2023-11-15T04:07:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 --2- 172.21.15.38:0/2715597070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b40fe750 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 -- 172.21.15.38:0/2715597070 >> 172.21.15.38:0/2715597070 conn(0x7f59b40f6250 msgr2=0x7f59b40f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 -- 172.21.15.38:0/2715597070 shutdown_connections 2023-11-15T04:07:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.250+0000 7f59b3fff700 1 -- 172.21.15.38:0/2715597070 wait complete. 2023-11-15T04:07:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.251+0000 7f59b3fff700 1 Processor -- start 2023-11-15T04:07:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.251+0000 7f59b3fff700 1 -- start start 2023-11-15T04:07:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.251+0000 7f59b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.251+0000 7f59b3fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b410a5b0 con 0x7f59b40704a0 2023-11-15T04:07:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60698/0 (socket says 172.21.15.38:60698) 2023-11-15T04:07:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b2ffd700 1 -- 172.21.15.38:0/3784772933 learned_addr learned my addr 172.21.15.38:0/3784772933 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b2ffd700 1 -- 172.21.15.38:0/3784772933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a400b200 con 0x7f59b40704a0 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b2ffd700 1 --2- 172.21.15.38:0/3784772933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59a4006c70 tx=0x7f59a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a4011b40 con 0x7f59b40704a0 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a4011ca0 con 0x7f59b40704a0 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b410a7b0 con 0x7f59b40704a0 2023-11-15T04:07:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a401d430 con 0x7f59b40704a0 2023-11-15T04:07:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.252+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b410abc0 con 0x7f59b40704a0 2023-11-15T04:07:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.253+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f59a400f070 con 0x7f59b40704a0 2023-11-15T04:07:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.253+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f59a4027460 con 0x7f59b40704a0 2023-11-15T04:07:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.253+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b4045ad0 con 0x7f59b40704a0 2023-11-15T04:07:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.256+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f59a4016070 con 0x7f59b40704a0 2023-11-15T04:07:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.328+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f59b4103c60 con 0x7f59b40704a0 2023-11-15T04:07:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.328+0000 7f599b7fe700 1 -- 172.21.15.38:0/3784772933 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f59a4036070 con 0x7f59b40704a0 2023-11-15T04:07:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: cluster: 2023-11-15T04:07:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: id: 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T04:07:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: services: 2023-11-15T04:07:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi038 (age 0.490215s) 2023-11-15T04:07:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T04:07:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T04:07:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: data: 2023-11-15T04:07:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T04:07:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: pgs: 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.329+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 msgr2=0x7f59b410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.329+0000 7f59b3fff700 1 --2- 172.21.15.38:0/3784772933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59a4006c70 tx=0x7f59a4016000).stop 2023-11-15T04:07:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.330+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 shutdown_connections 2023-11-15T04:07:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.330+0000 7f59b3fff700 1 --2- 172.21.15.38:0/3784772933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59b40704a0 0x7f59b410a0c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.330+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 >> 172.21.15.38:0/3784772933 conn(0x7f59b40f6250 msgr2=0x7f59b40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.330+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 shutdown_connections 2023-11-15T04:07:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:53.330+0000 7f59b3fff700 1 -- 172.21.15.38:0/3784772933 wait complete. 2023-11-15T04:07:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:mon is available 2023-11-15T04:07:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T04:07:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z -v /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/mon.smithi038:/var/lib/ceph/mon/ceph-smithi038:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi038/config 2023-11-15T04:07:53.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.839412+0000 mon.smithi038 (mon.0) 2 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2023-11-15T04:07:53.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.839755+0000 mon.smithi038 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} 2023-11-15T04:07:53.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.842085+0000 mon.smithi038 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T04:07:53.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.858259+0000 mon.smithi038 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:07:53.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: cluster 2023-11-15T04:07:52.858901+0000 mon.smithi038 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:07:53.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: audit 2023-11-15T04:07:53.329583+0000 mon.smithi038 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.38:0/3784772933' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T04:07:54.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.159+0000 7f8c35b18700 1 Processor -- start 2023-11-15T04:07:54.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.159+0000 7f8c35b18700 1 -- start start 2023-11-15T04:07:54.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c35b18700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:54.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c35b18700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300fdbc0 con 0x7f8c300fd2b0 2023-11-15T04:07:54.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:54.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60700/0 (socket says 172.21.15.38:60700) 2023-11-15T04:07:54.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2f7fe700 1 -- 172.21.15.38:0/3453915427 learned_addr learned my addr 172.21.15.38:0/3453915427 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:54.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2f7fe700 1 -- 172.21.15.38:0/3453915427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300fdd00 con 0x7f8c300fd2b0 2023-11-15T04:07:54.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2f7fe700 1 --2- 172.21.15.38:0/3453915427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8c2400b5f0 tx=0x7f8c24006ee0).ready entity=mon.0 client_cookie=4362fcc0a1f4dcdd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:54.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.160+0000 7f8c2e7fc700 1 -- 172.21.15.38:0/3453915427 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c2400d600 con 0x7f8c300fd2b0 2023-11-15T04:07:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c2e7fc700 1 -- 172.21.15.38:0/3453915427 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2400d760 con 0x7f8c300fd2b0 2023-11-15T04:07:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c2e7fc700 1 -- 172.21.15.38:0/3453915427 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c2400da30 con 0x7f8c300fd2b0 2023-11-15T04:07:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 -- 172.21.15.38:0/3453915427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 msgr2=0x7f8c300fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 --2- 172.21.15.38:0/3453915427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8c2400b5f0 tx=0x7f8c24006ee0).stop 2023-11-15T04:07:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 -- 172.21.15.38:0/3453915427 shutdown_connections 2023-11-15T04:07:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 --2- 172.21.15.38:0/3453915427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c300fd6d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 -- 172.21.15.38:0/3453915427 >> 172.21.15.38:0/3453915427 conn(0x7f8c300f5b10 msgr2=0x7f8c300f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 -- 172.21.15.38:0/3453915427 shutdown_connections 2023-11-15T04:07:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.161+0000 7f8c35b18700 1 -- 172.21.15.38:0/3453915427 wait complete. 2023-11-15T04:07:54.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c35b18700 1 Processor -- start 2023-11-15T04:07:54.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c35b18700 1 -- start start 2023-11-15T04:07:54.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c35b18700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:54.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c35b18700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3010a730 con 0x7f8c300fd2b0 2023-11-15T04:07:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60710/0 (socket says 172.21.15.38:60710) 2023-11-15T04:07:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.162+0000 7f8c2f7fe700 1 -- 172.21.15.38:0/2224631856 learned_addr learned my addr 172.21.15.38:0/2224631856 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c2f7fe700 1 -- 172.21.15.38:0/2224631856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2400b200 con 0x7f8c300fd2b0 2023-11-15T04:07:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c2f7fe700 1 --2- 172.21.15.38:0/2224631856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8c24006c70 tx=0x7f8c24016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:54.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c24011b40 con 0x7f8c300fd2b0 2023-11-15T04:07:54.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c24011ca0 con 0x7f8c300fd2b0 2023-11-15T04:07:54.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3010a930 con 0x7f8c300fd2b0 2023-11-15T04:07:54.171 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c2401d430 con 0x7f8c300fd2b0 2023-11-15T04:07:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3010ad40 con 0x7f8c300fd2b0 2023-11-15T04:07:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8c2400f070 con 0x7f8c300fd2b0 2023-11-15T04:07:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.163+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8c24027460 con 0x7f8c300fd2b0 2023-11-15T04:07:54.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.164+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c30045ad0 con 0x7f8c300fd2b0 2023-11-15T04:07:54.173 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.167+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8c24016070 con 0x7f8c300fd2b0 2023-11-15T04:07:54.238 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.237+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8c300f9dd0 con 0x7f8c300fd2b0 2023-11-15T04:07:54.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.241+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8c2401d590 con 0x7f8c300fd2b0 2023-11-15T04:07:54.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.241+0000 7f8c17fff700 1 -- 172.21.15.38:0/2224631856 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f8c240276f0 con 0x7f8c300fd2b0 2023-11-15T04:07:54.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [global] 2023-11-15T04:07:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: fsid = 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2023-11-15T04:07:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T04:07:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T04:07:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T04:07:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T04:07:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [mgr] 2023-11-15T04:07:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T04:07:54.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: [osd] 2023-11-15T04:07:54.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T04:07:54.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T04:07:54.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T04:07:54.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T04:07:54.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T04:07:54.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T04:07:54.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 msgr2=0x7f8c3010a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:54.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 --2- 172.21.15.38:0/2224631856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8c24006c70 tx=0x7f8c24016000).stop 2023-11-15T04:07:54.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 shutdown_connections 2023-11-15T04:07:54.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 --2- 172.21.15.38:0/2224631856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c300fd2b0 0x7f8c3010a240 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:54.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 >> 172.21.15.38:0/2224631856 conn(0x7f8c300f5b10 msgr2=0x7f8c300feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:54.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 shutdown_connections 2023-11-15T04:07:54.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:54.242+0000 7f8c35b18700 1 -- 172.21.15.38:0/2224631856 wait complete. 2023-11-15T04:07:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:Generating new minimal ceph.conf... 2023-11-15T04:07:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z -v /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/mon.smithi038:/var/lib/ceph/mon/ceph-smithi038:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi038/config 2023-11-15T04:07:55.005 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.001+0000 7f0ba5aaa700 1 Processor -- start 2023-11-15T04:07:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.001+0000 7f0ba5aaa700 1 -- start start 2023-11-15T04:07:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0ba5aaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0ba5aaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00fc770 con 0x7f0ba0070390 2023-11-15T04:07:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60712/0 (socket says 172.21.15.38:60712) 2023-11-15T04:07:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9f7fe700 1 -- 172.21.15.38:0/2553148852 learned_addr learned my addr 172.21.15.38:0/2553148852 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9f7fe700 1 -- 172.21.15.38:0/2553148852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba0101270 con 0x7f0ba0070390 2023-11-15T04:07:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9f7fe700 1 --2- 172.21.15.38:0/2553148852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0b9000b5f0 tx=0x7f0b90006ee0).ready entity=mon.0 client_cookie=abc6d0c1e44e5b78 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.002+0000 7f0b9e7fc700 1 -- 172.21.15.38:0/2553148852 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9000d600 con 0x7f0ba0070390 2023-11-15T04:07:55.008 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0b9e7fc700 1 -- 172.21.15.38:0/2553148852 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b9000dbd0 con 0x7f0ba0070390 2023-11-15T04:07:55.008 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0b9e7fc700 1 -- 172.21.15.38:0/2553148852 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b90011c00 con 0x7f0ba0070390 2023-11-15T04:07:55.008 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2553148852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 msgr2=0x7f0ba0100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:55.008 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 --2- 172.21.15.38:0/2553148852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0b9000b5f0 tx=0x7f0b90006ee0).stop 2023-11-15T04:07:55.008 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2553148852 shutdown_connections 2023-11-15T04:07:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 --2- 172.21.15.38:0/2553148852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba0100d80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2553148852 >> 172.21.15.38:0/2553148852 conn(0x7f0ba00f63c0 msgr2=0x7f0ba00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2553148852 shutdown_connections 2023-11-15T04:07:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.003+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2553148852 wait complete. 2023-11-15T04:07:55.011 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.004+0000 7f0ba5aaa700 1 Processor -- start 2023-11-15T04:07:55.011 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.004+0000 7f0ba5aaa700 1 -- start start 2023-11-15T04:07:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.004+0000 7f0ba5aaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.004+0000 7f0ba5aaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba010d200 con 0x7f0ba0070390 2023-11-15T04:07:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60728/0 (socket says 172.21.15.38:60728) 2023-11-15T04:07:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b9f7fe700 1 -- 172.21.15.38:0/2992963575 learned_addr learned my addr 172.21.15.38:0/2992963575 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b9f7fe700 1 -- 172.21.15.38:0/2992963575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9000b200 con 0x7f0ba0070390 2023-11-15T04:07:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b9f7fe700 1 --2- 172.21.15.38:0/2992963575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0b9000e040 tx=0x7f0b90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9000d9f0 con 0x7f0ba0070390 2023-11-15T04:07:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b9001a9f0 con 0x7f0ba0070390 2023-11-15T04:07:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba010d400 con 0x7f0ba0070390 2023-11-15T04:07:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9001b8c0 con 0x7f0ba0070390 2023-11-15T04:07:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.005+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba010d810 con 0x7f0ba0070390 2023-11-15T04:07:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.006+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0b9000f070 con 0x7f0ba0070390 2023-11-15T04:07:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.006+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0b900257d0 con 0x7f0ba0070390 2023-11-15T04:07:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.007+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba0045ad0 con 0x7f0ba0070390 2023-11-15T04:07:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.010+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0b90016070 con 0x7f0ba0070390 2023-11-15T04:07:55.080 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.079+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0ba010ec60 con 0x7f0ba0070390 2023-11-15T04:07:55.080 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.079+0000 7f0b97fff700 1 -- 172.21.15.38:0/2992963575 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f0b9001ba20 con 0x7f0ba0070390 2023-11-15T04:07:55.080 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 msgr2=0x7f0ba010cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 --2- 172.21.15.38:0/2992963575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0b9000e040 tx=0x7f0b90006ee0).stop 2023-11-15T04:07:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 shutdown_connections 2023-11-15T04:07:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 --2- 172.21.15.38:0/2992963575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ba0070390 0x7f0ba010cd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 >> 172.21.15.38:0/2992963575 conn(0x7f0ba00f63c0 msgr2=0x7f0ba00f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 shutdown_connections 2023-11-15T04:07:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:55.080+0000 7f0ba5aaa700 1 -- 172.21.15.38:0/2992963575 wait complete. 2023-11-15T04:07:55.276 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: audit 2023-11-15T04:07:54.238379+0000 mon.smithi038 (mon.0) 8 : audit [INF] from='client.? 172.21.15.38:0/2224631856' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:07:55.276 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: audit 2023-11-15T04:07:54.241884+0000 mon.smithi038 (mon.0) 9 : audit [INF] from='client.? 172.21.15.38:0/2224631856' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:07:55.444 INFO:teuthology.orchestra.run.smithi038.stderr:Restarting the monitor... 2023-11-15T04:07:55.445 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl restart ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:07:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:55 smithi038 systemd[1]: Stopping Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7... 2023-11-15T04:07:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: debug 2023-11-15T04:07:55.523+0000 7efe07df3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi038 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T04:07:55.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[111955]: debug 2023-11-15T04:07:55.523+0000 7efe07df3700 -1 mon.smithi038@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T04:07:56.253 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:55 smithi038 podman[112280]: ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038 2023-11-15T04:07:56.253 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 systemd[1]: ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service: Succeeded. 2023-11-15T04:07:56.253 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 systemd[1]: Stopped Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7. 2023-11-15T04:07:56.254 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 systemd[1]: Starting Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7... 2023-11-15T04:07:56.362 INFO:teuthology.orchestra.run.smithi038.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T04:07:56.362 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T04:07:56.735 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.733+0000 7f81ed7c9700 1 Processor -- start 2023-11-15T04:07:56.736 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81ed7c9700 1 -- start start 2023-11-15T04:07:56.736 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81ed7c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:56.736 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81ed7c9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80fb4a0 con 0x7f81e80fab90 2023-11-15T04:07:56.738 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:56.738 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60744/0 (socket says 172.21.15.38:60744) 2023-11-15T04:07:56.738 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.734+0000 7f81e6ffd700 1 -- 172.21.15.38:0/896770387 learned_addr learned my addr 172.21.15.38:0/896770387 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:56.738 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81e6ffd700 1 -- 172.21.15.38:0/896770387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80fb5e0 con 0x7f81e80fab90 2023-11-15T04:07:56.739 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81e6ffd700 1 --2- 172.21.15.38:0/896770387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f81d800b5f0 tx=0x7f81d8006ee0).ready entity=mon.0 client_cookie=6fe83ee3315211a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:56.739 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81e5ffb700 1 -- 172.21.15.38:0/896770387 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d800d600 con 0x7f81e80fab90 2023-11-15T04:07:56.739 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81e5ffb700 1 -- 172.21.15.38:0/896770387 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f81d800dbd0 con 0x7f81e80fab90 2023-11-15T04:07:56.739 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81e5ffb700 1 -- 172.21.15.38:0/896770387 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d8011c00 con 0x7f81e80fab90 2023-11-15T04:07:56.739 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81ed7c9700 1 -- 172.21.15.38:0/896770387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 msgr2=0x7f81e80fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:56.740 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81ed7c9700 1 --2- 172.21.15.38:0/896770387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f81d800b5f0 tx=0x7f81d8006ee0).stop 2023-11-15T04:07:56.740 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81ed7c9700 1 -- 172.21.15.38:0/896770387 shutdown_connections 2023-11-15T04:07:56.740 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81ed7c9700 1 --2- 172.21.15.38:0/896770387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e80fafb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.735+0000 7f81ed7c9700 1 -- 172.21.15.38:0/896770387 >> 172.21.15.38:0/896770387 conn(0x7f81e80f63b0 msgr2=0x7f81e80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.736+0000 7f81ed7c9700 1 -- 172.21.15.38:0/896770387 shutdown_connections 2023-11-15T04:07:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.736+0000 7f81ed7c9700 1 -- 172.21.15.38:0/896770387 wait complete. 2023-11-15T04:07:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.736+0000 7f81ed7c9700 1 Processor -- start 2023-11-15T04:07:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.736+0000 7f81ed7c9700 1 -- start start 2023-11-15T04:07:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81ed7c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81ed7c9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80704c0 con 0x7f81e80fab90 2023-11-15T04:07:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60750/0 (socket says 172.21.15.38:60750) 2023-11-15T04:07:56.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81e6ffd700 1 -- 172.21.15.38:0/3615338792 learned_addr learned my addr 172.21.15.38:0/3615338792 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:56.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81e6ffd700 1 -- 172.21.15.38:0/3615338792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d800b200 con 0x7f81e80fab90 2023-11-15T04:07:56.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81e6ffd700 1 --2- 172.21.15.38:0/3615338792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f81d800b1d0 tx=0x7f81d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:56.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d801b410 con 0x7f81e80fab90 2023-11-15T04:07:56.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f81d801b9e0 con 0x7f81e80fab90 2023-11-15T04:07:56.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d801a840 con 0x7f81e80fab90 2023-11-15T04:07:56.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e80706c0 con 0x7f81e80fab90 2023-11-15T04:07:56.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.737+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e810ddc0 con 0x7f81e80fab90 2023-11-15T04:07:56.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.738+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f81d800f070 con 0x7f81e80fab90 2023-11-15T04:07:56.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.738+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f81d8025b80 con 0x7f81e80fab90 2023-11-15T04:07:56.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.739+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e8045ad0 con 0x7f81e80fab90 2023-11-15T04:07:56.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.741+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f81d8016070 con 0x7f81e80fab90 2023-11-15T04:07:56.810 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.810+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f81e8027a00 con 0x7f81e80fab90 2023-11-15T04:07:56.818 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.817+0000 7f81cf7fe700 1 -- 172.21.15.38:0/3615338792 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f81d8025dc0 con 0x7f81e80fab90 2023-11-15T04:07:56.819 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 msgr2=0x7f81e810d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:56.819 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 --2- 172.21.15.38:0/3615338792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f81d800b1d0 tx=0x7f81d8006ee0).stop 2023-11-15T04:07:56.819 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 shutdown_connections 2023-11-15T04:07:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 --2- 172.21.15.38:0/3615338792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e80fab90 0x7f81e810d710 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 >> 172.21.15.38:0/3615338792 conn(0x7f81e80f63b0 msgr2=0x7f81e80f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:56.821 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 shutdown_connections 2023-11-15T04:07:56.821 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:56.818+0000 7f81ed7c9700 1 -- 172.21.15.38:0/3615338792 wait complete. 2023-11-15T04:07:56.840 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.354+0000 7f9ccd9d0700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:07:56.840 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.354+0000 7f9ccd9d0700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T04:07:56.840 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.354+0000 7f9ccd9d0700 0 pidfile_write: ignore empty --pid-file 2023-11-15T04:07:56.841 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 bash[112352]: 9f2c68e4246c942bf330648477dbc297c6acf1e4c2c2eaae41945263d2c80b85 2023-11-15T04:07:56.841 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.359+0000 7f9ccd9d0700 0 load: jerasure load: lrc load: isa 2023-11-15T04:07:56.841 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 systemd[1]: Started Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7. 2023-11-15T04:07:56.841 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:07:56.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:07:56.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:07:56.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: DB SUMMARY 2023-11-15T04:07:56.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T04:07:56.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T04:07:56.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T04:07:56.844 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.844 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi038/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T04:07:56.844 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.844 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi038/store.db: 000006.log size: 92918 ; 2023-11-15T04:07:56.845 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.845 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:07:56.845 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T04:07:56.845 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:07:56.845 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.env: 0x5624b59791c0 2023-11-15T04:07:56.846 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:07:56.846 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.info_log: 0x5624b64cf960 2023-11-15T04:07:56.846 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:07:56.846 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:07:56.846 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:07:56.847 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:07:56.847 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:07:56.848 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:07:56.848 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:07:56.848 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:07:56.848 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:07:56.849 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:07:56.849 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:07:56.849 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:07:56.849 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:07:56.849 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:07:56.850 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:07:56.850 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi038/store.db 2023-11-15T04:07:56.850 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:07:56.850 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:07:56.850 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:07:56.851 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:07:56.851 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:07:56.851 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:07:56.851 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:07:56.852 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:07:56.852 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:07:56.852 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:07:56.853 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.write_buffer_manager: 0x5624b7144030 2023-11-15T04:07:56.853 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:07:56.853 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:07:56.853 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:07:56.854 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:07:56.859 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:07:56.859 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:07:56.860 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:07:56.860 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:07:56.860 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:07:56.860 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:07:56.863 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:07:56.863 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:07:56.863 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:07:56.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:07:56.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.row_cache: None 2023-11-15T04:07:56.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:07:56.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:07:56.864 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:07:56.865 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:07:56.866 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:07:56.866 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:07:56.866 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:07:56.866 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:07:56.866 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:07:56.867 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:07:56.867 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:07:56.867 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:07:56.867 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:07:56.867 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:07:56.868 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:07:56.868 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:07:56.868 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:07:56.868 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:07:56.868 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:07:56.869 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:07:56.869 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:07:56.869 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:07:56.870 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:07:56.870 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:07:56.870 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:07:56.870 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:07:56.870 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:07:56.871 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:07:56.871 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:07:56.872 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:07:56.872 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:07:56.872 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.360+0000 7f9ccd9d0700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:07:56.872 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000005 2023-11-15T04:07:56.872 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.merge_operator: 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:07:56.873 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:07:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:07:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:07:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5624b63e6128) 2023-11-15T04:07:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cache_index_and_filter_blocks: 1 2023-11-15T04:07:56.874 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:07:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:07:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: pin_top_level_index_and_filter: 1 2023-11-15T04:07:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: index_type: 0 2023-11-15T04:07:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: data_block_index_type: 0 2023-11-15T04:07:56.875 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: index_shortening: 1 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: hash_index_allow_collision: 1 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: checksum: 1 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: no_block_cache: 0 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_cache: 0x5624b641cf10 2023-11-15T04:07:56.876 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_cache_name: BinnedLRUCache 2023-11-15T04:07:56.877 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_cache_options: 2023-11-15T04:07:56.877 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: capacity : 536870912 2023-11-15T04:07:56.877 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: num_shard_bits : 4 2023-11-15T04:07:56.877 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: strict_capacity_limit : 0 2023-11-15T04:07:56.877 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: high_pri_pool_ratio: 0.000 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_cache_compressed: (nil) 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: persistent_cache: (nil) 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_size: 4096 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_size_deviation: 10 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_restart_interval: 16 2023-11-15T04:07:56.878 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: index_block_restart_interval: 1 2023-11-15T04:07:56.879 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: metadata_block_size: 4096 2023-11-15T04:07:56.879 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: partition_filters: 0 2023-11-15T04:07:56.879 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: use_delta_encoding: 1 2023-11-15T04:07:56.880 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:07:56.880 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: whole_key_filtering: 1 2023-11-15T04:07:56.880 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: verify_compression: 0 2023-11-15T04:07:56.880 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: read_amp_bytes_per_bit: 0 2023-11-15T04:07:56.881 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: format_version: 2 2023-11-15T04:07:56.881 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: enable_index_compression: 1 2023-11-15T04:07:56.881 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: block_align: 0 2023-11-15T04:07:56.882 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.882 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:07:56.882 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:07:56.882 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:07:56.883 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:07:56.884 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:07:56.884 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:07:56.884 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:07:56.884 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:07:56.884 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:07:56.885 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:07:56.885 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:07:56.885 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:07:56.885 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:07:56.885 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:07:56.886 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:07:56.886 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:07:56.886 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:07:56.886 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:07:56.887 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:07:56.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:07:56.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:07:56.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:07:56.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:07:56.888 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:07:56.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:07:56.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:07:56.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:07:56.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:07:56.889 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:07:56.890 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:07:56.890 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:07:56.890 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:07:56.891 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:07:56.891 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:07:56.891 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:07:56.891 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:07:56.891 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:07:56.892 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:07:56.892 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:07:56.892 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:07:56.893 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:07:56.893 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:07:56.893 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:07:56.893 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:07:56.893 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:07:56.894 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:07:56.894 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:07:56.894 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:07:56.894 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:07:56.895 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:07:56.895 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:07:56.895 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:07:56.895 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:07:56.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:07:56.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:07:56.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:07:56.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:07:56.896 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.361+0000 7f9ccd9d0700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:07:56.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.362+0000 7f9ccd9d0700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi038/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:07:56.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.362+0000 7f9ccd9d0700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T04:07:56.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.897 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.362+0000 7f9ccd9d0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700021276363162, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T04:07:56.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.362+0000 7f9ccd9d0700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T04:07:56.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.362+0000 7f9ccd9d0700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T04:07:56.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.363+0000 7f9ccd9d0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700021276364314, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700021276, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T04:07:56.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.363+0000 7f9ccd9d0700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T04:07:56.898 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.899 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.368+0000 7f9ccd9d0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700021276369746, "job": 1, "event": "recovery_finished"} 2023-11-15T04:07:56.899 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.369+0000 7f9ccd9d0700 4 rocksdb: DB pointer 0x5624b64e1800 2023-11-15T04:07:56.899 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.370+0000 7f9cb47e8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:07:56.899 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.370+0000 7f9cb47e8700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:07:56.900 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** DB Stats ** 2023-11-15T04:07:56.900 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:56.900 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:07:56.901 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:07:56.901 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:07:56.901 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:07:56.901 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:07:56.901 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:07:56.902 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.902 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** Compaction Stats [default] ** 2023-11-15T04:07:56.902 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:56.903 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:56.903 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.903 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.903 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.903 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** Compaction Stats [default] ** 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:56.904 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:56.905 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:56.905 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:07:56.905 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:07:56.905 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:07:56.905 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Cumulative compaction: 0.00 GB write, 9.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:56.906 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Interval compaction: 0.00 GB write, 9.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:56.906 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:07:56.906 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.906 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:07:56.906 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** Compaction Stats [default] ** 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.907 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:07:56.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** Compaction Stats [default] ** 2023-11-15T04:07:56.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:07:56.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:07:56.908 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.6 0.00 0.00 1 0.001 0 0 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:07:56.909 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:07:56.910 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Cumulative compaction: 0.00 GB write, 9.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:56.910 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:07:56.910 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:07:56.910 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.911 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:07:56.911 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.911 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.370+0000 7f9ccd9d0700 0 starting mon.smithi038 rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi038 fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:56.911 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 1 mon.smithi038@-1(???) e1 preinit fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:07:56.912 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).mds e1 new map 2023-11-15T04:07:56.912 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).mds e1 print_map 2023-11-15T04:07:56.912 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: e1 2023-11-15T04:07:56.912 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T04:07:56.912 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T04:07:56.913 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: legacy client fscid: -1 2023-11-15T04:07:56.913 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.913 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: No filesystems configured 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 0 mon.smithi038@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:07:56.914 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:07:54.238379+0000 mon.smithi038 (mon.0) 8 : audit [INF] from='client.? 172.21.15.38:0/2224631856' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:07:56.915 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:07:54.241884+0000 mon.smithi038 (mon.0) 9 : audit [INF] from='client.? 172.21.15.38:0/2224631856' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:07:56.915 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.372+0000 7f9ccd9d0700 1 mon.smithi038@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T04:07:56.915 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expand map: {default=false} 2023-11-15T04:07:56.915 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta from 'false' to 'false' 2023-11-15T04:07:56.915 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expanded map: {default=false} 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expand map: {default=info} 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta from 'info' to 'info' 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expanded map: {default=info} 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expand map: {default=daemon} 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-15T04:07:56.916 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-15T04:07:56.917 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expand map: {default=debug} 2023-11-15T04:07:56.917 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-15T04:07:56.917 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:07:56.373+0000 7f9ccd9d0700 20 expand_channel_meta expanded map: {default=debug} 2023-11-15T04:07:56.917 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:07:56.390782+0000 mon.smithi038 (mon.0) 1 : cluster [INF] mon.smithi038 is new leader, mons smithi038 in quorum (ranks 0) 2023-11-15T04:07:56.917 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:07:56.390869+0000 mon.smithi038 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi038=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} 2023-11-15T04:07:56.918 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:07:56.390922+0000 mon.smithi038 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T04:07:56.918 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:07:56.390956+0000 mon.smithi038 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:07:56.918 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:07:56.391479+0000 mon.smithi038 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:07:57.186 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T04:07:57.186 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:07:57.186 INFO:teuthology.orchestra.run.smithi038.stderr:Creating mgr... 2023-11-15T04:07:57.187 INFO:teuthology.orchestra.run.smithi038.stderr:Verifying port 9283 ... 2023-11-15T04:07:57.189 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl daemon-reload 2023-11-15T04:07:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz 2023-11-15T04:07:57.328 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl reset-failed ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz 2023-11-15T04:07:57.334 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Failed to reset failed state of unit ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz.service: Unit ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz.service not loaded. 2023-11-15T04:07:57.335 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl enable ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz 2023-11-15T04:07:57.341 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: Created symlink /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.target.wants/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz.service → /etc/systemd/system/ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:07:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mgr.smithi038.bnbogz 2023-11-15T04:07:57.822 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:07:57.833 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2023-11-15T04:07:57.833 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:07:57.842 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:07:57.843 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2023-11-15T04:07:57.843 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:07:57.844 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:07:57.844 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:07:57.852 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2023-11-15T04:07:57.853 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:07:57.861 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:07:57.862 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2023-11-15T04:07:57.862 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T04:07:57.862 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr to start... 2023-11-15T04:07:57.863 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr... 2023-11-15T04:07:57.863 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:07:58.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:07:56.818314+0000 mon.smithi038 (mon.0) 6 : audit [INF] from='client.? 172.21.15.38:0/3615338792' entity='client.admin' 2023-11-15T04:07:58.256 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.253+0000 7fdf122ef700 1 Processor -- start 2023-11-15T04:07:58.256 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.253+0000 7fdf122ef700 1 -- start start 2023-11-15T04:07:58.256 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf122ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:58.256 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf122ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0fb490 con 0x7fdf0c0fab80 2023-11-15T04:07:58.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:58.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55412/0 (socket says 172.21.15.38:55412) 2023-11-15T04:07:58.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf0bfff700 1 -- 172.21.15.38:0/2789926988 learned_addr learned my addr 172.21.15.38:0/2789926988 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:58.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf0bfff700 1 -- 172.21.15.38:0/2789926988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0fb5d0 con 0x7fdf0c0fab80 2023-11-15T04:07:58.259 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.254+0000 7fdf0bfff700 1 --2- 172.21.15.38:0/2789926988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fdefc00b5f0 tx=0x7fdefc006ee0).ready entity=mon.0 client_cookie=aeeb901e1ad9c349 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:58.259 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.255+0000 7fdf0affd700 1 -- 172.21.15.38:0/2789926988 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdefc00d600 con 0x7fdf0c0fab80 2023-11-15T04:07:58.260 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.255+0000 7fdf0affd700 1 -- 172.21.15.38:0/2789926988 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdefc00dbd0 con 0x7fdf0c0fab80 2023-11-15T04:07:58.260 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.255+0000 7fdf0affd700 1 -- 172.21.15.38:0/2789926988 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdefc011c00 con 0x7fdf0c0fab80 2023-11-15T04:07:58.260 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 -- 172.21.15.38:0/2789926988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 msgr2=0x7fdf0c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:58.260 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 --2- 172.21.15.38:0/2789926988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fdefc00b5f0 tx=0x7fdefc006ee0).stop 2023-11-15T04:07:58.261 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 -- 172.21.15.38:0/2789926988 shutdown_connections 2023-11-15T04:07:58.261 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 --2- 172.21.15.38:0/2789926988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c0fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:58.261 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 -- 172.21.15.38:0/2789926988 >> 172.21.15.38:0/2789926988 conn(0x7fdf0c0f63c0 msgr2=0x7fdf0c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:58.261 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 -- 172.21.15.38:0/2789926988 shutdown_connections 2023-11-15T04:07:58.261 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.256+0000 7fdf122ef700 1 -- 172.21.15.38:0/2789926988 wait complete. 2023-11-15T04:07:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf122ef700 1 Processor -- start 2023-11-15T04:07:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf122ef700 1 -- start start 2023-11-15T04:07:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf122ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:07:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf122ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0ff290 con 0x7fdf0c0fab80 2023-11-15T04:07:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:07:58.263 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55426/0 (socket says 172.21.15.38:55426) 2023-11-15T04:07:58.263 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.257+0000 7fdf0bfff700 1 -- 172.21.15.38:0/766007542 learned_addr learned my addr 172.21.15.38:0/766007542 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:07:58.263 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf0bfff700 1 -- 172.21.15.38:0/766007542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc00b200 con 0x7fdf0c0fab80 2023-11-15T04:07:58.263 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf0bfff700 1 --2- 172.21.15.38:0/766007542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdefc00b1d0 tx=0x7fdefc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:07:58.263 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdefc01b410 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdefc01b9e0 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c0ff430 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdefc01a840 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c0ff840 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdefc00f070 con 0x7fdf0c0fab80 2023-11-15T04:07:58.264 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.258+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdefc025c90 con 0x7fdf0c0fab80 2023-11-15T04:07:58.265 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.260+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c045ad0 con 0x7fdf0c0fab80 2023-11-15T04:07:58.265 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.262+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdefc016070 con 0x7fdf0c0fab80 2023-11-15T04:07:58.336 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.335+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdf0c0008d0 con 0x7fdf0c0fab80 2023-11-15T04:07:58.337 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.335+0000 7fdf08ff9700 1 -- 172.21.15.38:0/766007542 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fdefc01bb50 con 0x7fdf0c0fab80 2023-11-15T04:07:58.337 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.337 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:07:58.337 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "7bd2f4c8-836c-11ee-8dbe-212e2dc638e7", 2023-11-15T04:07:58.337 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:07:58.338 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:07:58.338 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:07:58.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:07:58.343 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:07:58.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:07:58.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2023-11-15T04:07:58.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:07:58.344 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:07:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2023-11-15T04:07:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:07:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T04:07:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:07:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:07:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:07:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:07:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:07:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:07:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:07:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:07:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:07:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:07:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:07:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:07:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:07:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:07:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:07:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:07:58.351 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:07:58.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:07:58.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:07:58.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:07:58.352 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:07:52.840790+0000", 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:07:58.353 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.336+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 msgr2=0x7fdf0c100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:07:58.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.336+0000 7fdf122ef700 1 --2- 172.21.15.38:0/766007542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdefc00b1d0 tx=0x7fdefc006ee0).stop 2023-11-15T04:07:58.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.336+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 shutdown_connections 2023-11-15T04:07:58.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.336+0000 7fdf122ef700 1 --2- 172.21.15.38:0/766007542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0c0fab80 0x7fdf0c100e10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:07:58.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.336+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 >> 172.21.15.38:0/766007542 conn(0x7fdf0c0f63c0 msgr2=0x7fdf0c070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:07:58.354 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.337+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 shutdown_connections 2023-11-15T04:07:58.355 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:07:58.337+0000 7fdf122ef700 1 -- 172.21.15.38:0/766007542 wait complete. 2023-11-15T04:07:58.727 INFO:teuthology.orchestra.run.smithi038.stderr:mgr not available, waiting (1/15)... 2023-11-15T04:07:59.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:07:58 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:07:58.336496+0000 mon.smithi038 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.38:0/766007542' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:08:00.729 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:08:01.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.152+0000 7f97287bb700 1 Processor -- start 2023-11-15T04:08:01.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.152+0000 7f97287bb700 1 -- start start 2023-11-15T04:08:01.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.152+0000 7f97287bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:01.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.152+0000 7f97287bb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97200fc750 con 0x7f97200703b0 2023-11-15T04:08:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9726557700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9726557700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55428/0 (socket says 172.21.15.38:55428) 2023-11-15T04:08:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9726557700 1 -- 172.21.15.38:0/3719628293 learned_addr learned my addr 172.21.15.38:0/3719628293 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9726557700 1 -- 172.21.15.38:0/3719628293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9720101250 con 0x7f97200703b0 2023-11-15T04:08:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9726557700 1 --2- 172.21.15.38:0/3719628293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f971c00b5f0 tx=0x7f971c006ee0).ready entity=mon.0 client_cookie=a25eb9cae87707c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9725555700 1 -- 172.21.15.38:0/3719628293 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f971c00d600 con 0x7f97200703b0 2023-11-15T04:08:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9725555700 1 -- 172.21.15.38:0/3719628293 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f971c00dbd0 con 0x7f97200703b0 2023-11-15T04:08:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.153+0000 7f9725555700 1 -- 172.21.15.38:0/3719628293 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f971c011c00 con 0x7f97200703b0 2023-11-15T04:08:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.154+0000 7f97287bb700 1 -- 172.21.15.38:0/3719628293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 msgr2=0x7f9720100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.154+0000 7f97287bb700 1 --2- 172.21.15.38:0/3719628293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f971c00b5f0 tx=0x7f971c006ee0).stop 2023-11-15T04:08:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.154+0000 7f97287bb700 1 -- 172.21.15.38:0/3719628293 shutdown_connections 2023-11-15T04:08:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.154+0000 7f97287bb700 1 --2- 172.21.15.38:0/3719628293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f9720100d60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.154+0000 7f97287bb700 1 -- 172.21.15.38:0/3719628293 >> 172.21.15.38:0/3719628293 conn(0x7f97200f63c0 msgr2=0x7f97200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.155+0000 7f97287bb700 1 -- 172.21.15.38:0/3719628293 shutdown_connections 2023-11-15T04:08:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.155+0000 7f97287bb700 1 -- 172.21.15.38:0/3719628293 wait complete. 2023-11-15T04:08:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.155+0000 7f97287bb700 1 Processor -- start 2023-11-15T04:08:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.155+0000 7f97287bb700 1 -- start start 2023-11-15T04:08:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f97287bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f97287bb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972010d1e0 con 0x7f97200703b0 2023-11-15T04:08:01.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f9726557700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:01.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f9726557700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55442/0 (socket says 172.21.15.38:55442) 2023-11-15T04:08:01.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f9726557700 1 -- 172.21.15.38:0/3149001118 learned_addr learned my addr 172.21.15.38:0/3149001118 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:01.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f9726557700 1 -- 172.21.15.38:0/3149001118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c00b200 con 0x7f97200703b0 2023-11-15T04:08:01.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.156+0000 7f9726557700 1 --2- 172.21.15.38:0/3149001118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f971c00e040 tx=0x7f971c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:01.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f971c00d9f0 con 0x7f97200703b0 2023-11-15T04:08:01.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f971c01a9f0 con 0x7f97200703b0 2023-11-15T04:08:01.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972010d3e0 con 0x7f97200703b0 2023-11-15T04:08:01.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f971c01b8c0 con 0x7f97200703b0 2023-11-15T04:08:01.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972010d7f0 con 0x7f97200703b0 2023-11-15T04:08:01.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f971c00f070 con 0x7f97200703b0 2023-11-15T04:08:01.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.157+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f971c025b20 con 0x7f97200703b0 2023-11-15T04:08:01.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.158+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9720045ad0 con 0x7f97200703b0 2023-11-15T04:08:01.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.160+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f971c016070 con 0x7f97200703b0 2023-11-15T04:08:01.236 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.235+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f972010e6f0 con 0x7f97200703b0 2023-11-15T04:08:01.236 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.235+0000 7f970effd700 1 -- 172.21.15.38:0/3149001118 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f971c01bd20 con 0x7f97200703b0 2023-11-15T04:08:01.236 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:01.237 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "7bd2f4c8-836c-11ee-8dbe-212e2dc638e7", 2023-11-15T04:08:01.237 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:08:01.237 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:08:01.243 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:08:01.243 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:08:01.243 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.243 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:08:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:08:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2023-11-15T04:08:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:08:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:08:01.245 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:08:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:08:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:08:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:08:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:08:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:08:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:08:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:08:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:08:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:08:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:08:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:08:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:08:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:08:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:08:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:08:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:08:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:08:01.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:01.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:08:01.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:08:01.250 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:08:01.251 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:08:01.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:01.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:01.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:08:01.252 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:07:52.840790+0000", 2023-11-15T04:08:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:08:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:01.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 msgr2=0x7f972010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:01.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 --2- 172.21.15.38:0/3149001118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f971c00e040 tx=0x7f971c006ee0).stop 2023-11-15T04:08:01.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 shutdown_connections 2023-11-15T04:08:01.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 --2- 172.21.15.38:0/3149001118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97200703b0 0x7f972010ccf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:01.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 >> 172.21.15.38:0/3149001118 conn(0x7f97200f63c0 msgr2=0x7f97200f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:01.255 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 shutdown_connections 2023-11-15T04:08:01.255 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:01.237+0000 7f97287bb700 1 -- 172.21.15.38:0/3149001118 wait complete. 2023-11-15T04:08:01.562 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:01.236218+0000 mon.smithi038 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.38:0/3149001118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:08:01.651 INFO:teuthology.orchestra.run.smithi038.stderr:mgr not available, waiting (2/15)... 2023-11-15T04:08:03.653 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:08:03.826 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:03.495139+0000 mon.smithi038 (mon.0) 9 : cluster [INF] Activating manager daemon smithi038.bnbogz 2023-11-15T04:08:03.827 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:03.498190+0000 mon.smithi038 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi038.bnbogz(active, starting, since 0.00317465s) 2023-11-15T04:08:03.827 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.499394+0000 mon.smithi038 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:08:03.827 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.499556+0000 mon.smithi038 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:08:03.827 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.499689+0000 mon.smithi038 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:08:03.827 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.499835+0000 mon.smithi038 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2023-11-15T04:08:03.828 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.500246+0000 mon.smithi038 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mgr metadata", "who": "smithi038.bnbogz", "id": "smithi038.bnbogz"}]: dispatch 2023-11-15T04:08:03.828 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:03.513755+0000 mon.smithi038 (mon.0) 16 : cluster [INF] Manager daemon smithi038.bnbogz is now available 2023-11-15T04:08:03.828 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.526651+0000 mon.smithi038 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:03.828 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.528435+0000 mon.smithi038 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:08:03.828 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.529848+0000 mon.smithi038 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:08:03.829 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.540373+0000 mon.smithi038 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:04.048 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.046+0000 7f877e70f700 1 Processor -- start 2023-11-15T04:08:04.048 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.046+0000 7f877e70f700 1 -- start start 2023-11-15T04:08:04.048 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.047+0000 7f877e70f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:04.048 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.047+0000 7f877e70f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780fedf0 con 0x7f8778070410 2023-11-15T04:08:04.050 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.047+0000 7f8777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:04.051 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.047+0000 7f8777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55482/0 (socket says 172.21.15.38:55482) 2023-11-15T04:08:04.051 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.047+0000 7f8777fff700 1 -- 172.21.15.38:0/3397986469 learned_addr learned my addr 172.21.15.38:0/3397986469 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:04.051 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.048+0000 7f8777fff700 1 -- 172.21.15.38:0/3397986469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87780fef30 con 0x7f8778070410 2023-11-15T04:08:04.051 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.048+0000 7f8777fff700 1 --2- 172.21.15.38:0/3397986469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f876800b5f0 tx=0x7f8768006ee0).ready entity=mon.0 client_cookie=8c3a631a699a973 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:04.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.048+0000 7f8776ffd700 1 -- 172.21.15.38:0/3397986469 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876800d600 con 0x7f8778070410 2023-11-15T04:08:04.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.048+0000 7f8776ffd700 1 -- 172.21.15.38:0/3397986469 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f876800dbd0 con 0x7f8778070410 2023-11-15T04:08:04.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.048+0000 7f8776ffd700 1 -- 172.21.15.38:0/3397986469 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8768011c00 con 0x7f8778070410 2023-11-15T04:08:04.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 -- 172.21.15.38:0/3397986469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 msgr2=0x7f87780fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:04.052 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 --2- 172.21.15.38:0/3397986469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f876800b5f0 tx=0x7f8768006ee0).stop 2023-11-15T04:08:04.053 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 -- 172.21.15.38:0/3397986469 shutdown_connections 2023-11-15T04:08:04.053 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 --2- 172.21.15.38:0/3397986469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f87780fe900 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:04.053 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 -- 172.21.15.38:0/3397986469 >> 172.21.15.38:0/3397986469 conn(0x7f87780f63c0 msgr2=0x7f87780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:04.053 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 -- 172.21.15.38:0/3397986469 shutdown_connections 2023-11-15T04:08:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.049+0000 7f877e70f700 1 -- 172.21.15.38:0/3397986469 wait complete. 2023-11-15T04:08:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.050+0000 7f877e70f700 1 Processor -- start 2023-11-15T04:08:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.050+0000 7f877e70f700 1 -- start start 2023-11-15T04:08:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f877e70f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f877e70f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877810aeb0 con 0x7f8778070410 2023-11-15T04:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55486/0 (socket says 172.21.15.38:55486) 2023-11-15T04:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8777fff700 1 -- 172.21.15.38:0/1409733176 learned_addr learned my addr 172.21.15.38:0/1409733176 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8777fff700 1 -- 172.21.15.38:0/1409733176 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876800b200 con 0x7f8778070410 2023-11-15T04:08:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8777fff700 1 --2- 172.21.15.38:0/1409733176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f876800b190 tx=0x7f8768006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876801b410 con 0x7f8778070410 2023-11-15T04:08:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f876801b9e0 con 0x7f8778070410 2023-11-15T04:08:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876801a880 con 0x7f8778070410 2023-11-15T04:08:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877810b0b0 con 0x7f8778070410 2023-11-15T04:08:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.051+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877810b4c0 con 0x7f8778070410 2023-11-15T04:08:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.053+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f876800f070 con 0x7f8778070410 2023-11-15T04:08:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.053+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8778045ad0 con 0x7f8778070410 2023-11-15T04:08:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.053+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8768041030 con 0x7f8778070410 2023-11-15T04:08:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.055+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8768022760 con 0x7f8778070410 2023-11-15T04:08:04.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.127+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f87780ffbb0 con 0x7f8778070410 2023-11-15T04:08:04.129 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f8774ff9700 1 -- 172.21.15.38:0/1409733176 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f8768022350 con 0x7f8778070410 2023-11-15T04:08:04.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:04.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "7bd2f4c8-836c-11ee-8dbe-212e2dc638e7", 2023-11-15T04:08:04.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:08:04.130 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:08:04.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:08:04.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:08:04.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.131 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:08:04.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:08:04.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2023-11-15T04:08:04.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:04.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:08:04.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2023-11-15T04:08:04.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:04.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T04:08:04.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:08:04.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:04.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:08:04.139 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:08:04.139 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.139 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:08:04.140 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:04.140 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:08:04.140 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:08:04.140 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:08:04.140 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:08:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:08:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:08:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:08:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:08:04.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:08:04.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:08:04.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:08:04.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:04.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:08:04.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:08:04.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:08:04.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:08:04.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:08:04.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:08:04.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:08:04.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:08:04.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:04.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:08:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:07:52.840790+0000", 2023-11-15T04:08:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 msgr2=0x7f877810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f877e70f700 1 --2- 172.21.15.38:0/1409733176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f876800b190 tx=0x7f8768006ee0).stop 2023-11-15T04:08:04.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 shutdown_connections 2023-11-15T04:08:04.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f877e70f700 1 --2- 172.21.15.38:0/1409733176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8778070410 0x7f877810a9c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:04.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.128+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 >> 172.21.15.38:0/1409733176 conn(0x7f87780f63c0 msgr2=0x7f87780f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:04.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.129+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 shutdown_connections 2023-11-15T04:08:04.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:04.129+0000 7f877e70f700 1 -- 172.21.15.38:0/1409733176 wait complete. 2023-11-15T04:08:04.477 INFO:teuthology.orchestra.run.smithi038.stderr:mgr not available, waiting (3/15)... 2023-11-15T04:08:04.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:03.549828+0000 mon.smithi038 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.38:0/1613692097' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:04.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:04.128934+0000 mon.smithi038 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.38:0/1409733176' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:08:04.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:04.506643+0000 mon.smithi038 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi038.bnbogz(active, since 1.01163s) 2023-11-15T04:08:06.479 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:08:06.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:06 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:05.558735+0000 mon.smithi038 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi038.bnbogz(active, since 2s) 2023-11-15T04:08:06.898 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.896+0000 7f9870512700 1 Processor -- start 2023-11-15T04:08:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f9870512700 1 -- start start 2023-11-15T04:08:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f9870512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f9870512700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680fcba0 con 0x7f98680fc290 2023-11-15T04:08:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f986e2ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f986e2ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55500/0 (socket says 172.21.15.38:55500) 2023-11-15T04:08:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f986e2ae700 1 -- 172.21.15.38:0/3163108804 learned_addr learned my addr 172.21.15.38:0/3163108804 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.897+0000 7f986e2ae700 1 -- 172.21.15.38:0/3163108804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680fcce0 con 0x7f98680fc290 2023-11-15T04:08:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f986e2ae700 1 --2- 172.21.15.38:0/3163108804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f986400b5f0 tx=0x7f9864006ee0).ready entity=mon.0 client_cookie=3ee3964a23b254d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f986d2ac700 1 -- 172.21.15.38:0/3163108804 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986400d600 con 0x7f98680fc290 2023-11-15T04:08:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f986d2ac700 1 -- 172.21.15.38:0/3163108804 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f986400dbd0 con 0x7f98680fc290 2023-11-15T04:08:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f986d2ac700 1 -- 172.21.15.38:0/3163108804 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9864011c00 con 0x7f98680fc290 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f9870512700 1 -- 172.21.15.38:0/3163108804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 msgr2=0x7f98680fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f9870512700 1 --2- 172.21.15.38:0/3163108804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f986400b5f0 tx=0x7f9864006ee0).stop 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f9870512700 1 -- 172.21.15.38:0/3163108804 shutdown_connections 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f9870512700 1 --2- 172.21.15.38:0/3163108804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f98680fc6b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.898+0000 7f9870512700 1 -- 172.21.15.38:0/3163108804 >> 172.21.15.38:0/3163108804 conn(0x7f98680f5b40 msgr2=0x7f98680f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 -- 172.21.15.38:0/3163108804 shutdown_connections 2023-11-15T04:08:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 -- 172.21.15.38:0/3163108804 wait complete. 2023-11-15T04:08:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 Processor -- start 2023-11-15T04:08:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 -- start start 2023-11-15T04:08:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.899+0000 7f9870512700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986810d0e0 con 0x7f98680fc290 2023-11-15T04:08:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f986e2ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f986e2ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55504/0 (socket says 172.21.15.38:55504) 2023-11-15T04:08:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f986e2ae700 1 -- 172.21.15.38:0/1492778060 learned_addr learned my addr 172.21.15.38:0/1492778060 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f986e2ae700 1 -- 172.21.15.38:0/1492778060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986400b200 con 0x7f98680fc290 2023-11-15T04:08:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f986e2ae700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f986400e040 tx=0x7f9864006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986401b410 con 0x7f98680fc290 2023-11-15T04:08:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986810d2e0 con 0x7f98680fc290 2023-11-15T04:08:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f986401b9e0 con 0x7f98680fc290 2023-11-15T04:08:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986810d6f0 con 0x7f98680fc290 2023-11-15T04:08:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.900+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986401a820 con 0x7f98680fc290 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.901+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f986400f070 con 0x7f98680fc290 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.901+0000 7f9856ffd700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 0x7f98500321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.901+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9864042150 con 0x7f98680fc290 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.901+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9868045ad0 con 0x7f98680fc290 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.901+0000 7f986daad700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 0x7f98500321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.902+0000 7f986daad700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 0x7f98500321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f98580096a0 tx=0x7f9858006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:06.906+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9864016070 con 0x7f98680fc290 2023-11-15T04:08:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.132+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9868101e00 con 0x7f98680fc290 2023-11-15T04:08:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.132+0000 7f9856ffd700 1 -- 172.21.15.38:0/1492778060 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9864025460 con 0x7f98680fc290 2023-11-15T04:08:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsid": "7bd2f4c8-836c-11ee-8dbe-212e2dc638e7", 2023-11-15T04:08:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:08:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:08:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:08:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:08:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:08:07.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:08:07.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 0 2023-11-15T04:08:07.142 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "smithi038" 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:08:07.143 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:07.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:08:07.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:08:07.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:08:07.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:08:07.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:08:07.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:08:07.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:08:07.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:08:07.146 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:08:07.147 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:08:07.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:07.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:08:07.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:08:07.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:08:07.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:08:07.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:08:07.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:08:07.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:08:07.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:08:07.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ], 2023-11-15T04:08:07.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:07.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:08:07.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:08:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:07:52.840790+0000", 2023-11-15T04:08:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:08:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: }, 2023-11-15T04:08:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:08:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 msgr2=0x7f98500321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 0x7f98500321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f98580096a0 tx=0x7f9858006b40).stop 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 msgr2=0x7f986810cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f986400e040 tx=0x7f9864006ee0).stop 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 shutdown_connections 2023-11-15T04:08:07.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f985002fd00 0x7f98500321c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:07.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 --2- 172.21.15.38:0/1492778060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98680fc290 0x7f986810cbf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:07.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 >> 172.21.15.38:0/1492778060 conn(0x7f98680f5b40 msgr2=0x7f98680f7640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:07.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 shutdown_connections 2023-11-15T04:08:07.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.134+0000 7f9870512700 1 -- 172.21.15.38:0/1492778060 wait complete. 2023-11-15T04:08:07.493 INFO:teuthology.orchestra.run.smithi038.stderr:mgr is available 2023-11-15T04:08:07.493 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling cephadm module... 2023-11-15T04:08:07.494 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T04:08:07.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:07.133557+0000 mon.smithi038 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.38:0/1492778060' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:08:07.846 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.845+0000 7ff6bfb71700 1 Processor -- start 2023-11-15T04:08:07.848 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.845+0000 7ff6bfb71700 1 -- start start 2023-11-15T04:08:07.848 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.845+0000 7ff6bfb71700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:07.848 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.845+0000 7ff6bfb71700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b80fc5b0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bd90d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:07.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bd90d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35114/0 (socket says 172.21.15.38:35114) 2023-11-15T04:08:07.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bd90d700 1 -- 172.21.15.38:0/887089616 learned_addr learned my addr 172.21.15.38:0/887089616 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:07.849 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bd90d700 1 -- 172.21.15.38:0/887089616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b80fc6f0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.850 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bd90d700 1 --2- 172.21.15.38:0/887089616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff6ac00b5f0 tx=0x7ff6ac006ee0).ready entity=mon.0 client_cookie=995cbe9e4a5c9e2c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:07.850 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bc90b700 1 -- 172.21.15.38:0/887089616 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac00d600 con 0x7ff6b80fbca0 2023-11-15T04:08:07.850 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bc90b700 1 -- 172.21.15.38:0/887089616 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff6ac00dbd0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.850 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.846+0000 7ff6bc90b700 1 -- 172.21.15.38:0/887089616 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac011c00 con 0x7ff6b80fbca0 2023-11-15T04:08:07.850 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 -- 172.21.15.38:0/887089616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 msgr2=0x7ff6b80fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/887089616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff6ac00b5f0 tx=0x7ff6ac006ee0).stop 2023-11-15T04:08:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 -- 172.21.15.38:0/887089616 shutdown_connections 2023-11-15T04:08:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/887089616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b80fc0c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 -- 172.21.15.38:0/887089616 >> 172.21.15.38:0/887089616 conn(0x7ff6b80f74c0 msgr2=0x7ff6b80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 -- 172.21.15.38:0/887089616 shutdown_connections 2023-11-15T04:08:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.847+0000 7ff6bfb71700 1 -- 172.21.15.38:0/887089616 wait complete. 2023-11-15T04:08:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bfb71700 1 Processor -- start 2023-11-15T04:08:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bfb71700 1 -- start start 2023-11-15T04:08:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bfb71700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bfb71700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b810d1e0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bd90d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:07.855 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bd90d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35130/0 (socket says 172.21.15.38:35130) 2023-11-15T04:08:07.855 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.848+0000 7ff6bd90d700 1 -- 172.21.15.38:0/1197211574 learned_addr learned my addr 172.21.15.38:0/1197211574 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:07.855 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6bd90d700 1 -- 172.21.15.38:0/1197211574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac00b200 con 0x7ff6b80fbca0 2023-11-15T04:08:07.855 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6bd90d700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff6ac00e040 tx=0x7ff6ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:07.855 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac01b410 con 0x7ff6b80fbca0 2023-11-15T04:08:07.856 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff6ac01b9e0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.856 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b810d3e0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.856 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac01a820 con 0x7ff6b80fbca0 2023-11-15T04:08:07.856 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.849+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b810d7f0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.856 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7ff6ac00f070 con 0x7ff6b80fbca0 2023-11-15T04:08:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6aa7fc700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 0x7ff6a0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff6ac041270 con 0x7ff6b80fbca0 2023-11-15T04:08:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6bd10c700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 0x7ff6a0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b8045ad0 con 0x7ff6b80fbca0 2023-11-15T04:08:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.850+0000 7ff6bd10c700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 0x7ff6a0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff6b40096a0 tx=0x7ff6b4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:07.858 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:07.855+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6ac016070 con 0x7ff6b80fbca0 2023-11-15T04:08:08.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.060+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff6b80fd9b0 con 0x7ff6b80fbca0 2023-11-15T04:08:08.578 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.577+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff6ac01bb50 con 0x7ff6b80fbca0 2023-11-15T04:08:08.579 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.578+0000 7ff6aa7fc700 1 -- 172.21.15.38:0/1197211574 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff6ac023940 con 0x7ff6b80fbca0 2023-11-15T04:08:08.581 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 msgr2=0x7ff6a0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:08.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 0x7ff6a0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff6b40096a0 tx=0x7ff6b4006b40).stop 2023-11-15T04:08:08.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 msgr2=0x7ff6b810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:08.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff6ac00e040 tx=0x7ff6ac006ee0).stop 2023-11-15T04:08:08.583 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 shutdown_connections 2023-11-15T04:08:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7ff6a002fd50 0x7ff6a0032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 --2- 172.21.15.38:0/1197211574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6b80fbca0 0x7ff6b810ccf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 >> 172.21.15.38:0/1197211574 conn(0x7ff6b80f74c0 msgr2=0x7ff6b80f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 shutdown_connections 2023-11-15T04:08:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:08.581+0000 7ff6bfb71700 1 -- 172.21.15.38:0/1197211574 wait complete. 2023-11-15T04:08:08.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:08 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:08.062169+0000 mon.smithi038 (mon.0) 26 : audit [INF] from='client.? 172.21.15.38:0/1197211574' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T04:08:09.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:08.579620+0000 mon.smithi038 (mon.0) 27 : audit [INF] from='client.? 172.21.15.38:0/1197211574' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T04:08:09.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:08.579680+0000 mon.smithi038 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi038.bnbogz(active, since 5s) 2023-11-15T04:08:10.266 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:08:10.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.758+0000 7f4c1897a700 1 Processor -- start 2023-11-15T04:08:10.761 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.758+0000 7f4c1897a700 1 -- start start 2023-11-15T04:08:10.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1897a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:10.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1897a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140fb4b0 con 0x7f4c140faba0 2023-11-15T04:08:10.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:10.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35160/0 (socket says 172.21.15.38:35160) 2023-11-15T04:08:10.762 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1259c700 1 -- 172.21.15.38:0/2093044610 learned_addr learned my addr 172.21.15.38:0/2093044610 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:10.763 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1259c700 1 -- 172.21.15.38:0/2093044610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140fb5f0 con 0x7f4c140faba0 2023-11-15T04:08:10.763 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.759+0000 7f4c1259c700 1 --2- 172.21.15.38:0/2093044610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4c0400b5f0 tx=0x7f4c04006ee0).ready entity=mon.0 client_cookie=76610dd5622401e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:10.763 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1159a700 1 -- 172.21.15.38:0/2093044610 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0400d600 con 0x7f4c140faba0 2023-11-15T04:08:10.763 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1159a700 1 -- 172.21.15.38:0/2093044610 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4c0400dbd0 con 0x7f4c140faba0 2023-11-15T04:08:10.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1159a700 1 -- 172.21.15.38:0/2093044610 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c04011c00 con 0x7f4c140faba0 2023-11-15T04:08:10.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1897a700 1 -- 172.21.15.38:0/2093044610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 msgr2=0x7f4c140fafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:10.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1897a700 1 --2- 172.21.15.38:0/2093044610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4c0400b5f0 tx=0x7f4c04006ee0).stop 2023-11-15T04:08:10.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1897a700 1 -- 172.21.15.38:0/2093044610 shutdown_connections 2023-11-15T04:08:10.764 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1897a700 1 --2- 172.21.15.38:0/2093044610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c140fafc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:10.765 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.760+0000 7f4c1897a700 1 -- 172.21.15.38:0/2093044610 >> 172.21.15.38:0/2093044610 conn(0x7f4c140f63c0 msgr2=0x7f4c140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:10.765 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 -- 172.21.15.38:0/2093044610 shutdown_connections 2023-11-15T04:08:10.765 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 -- 172.21.15.38:0/2093044610 wait complete. 2023-11-15T04:08:10.768 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 Processor -- start 2023-11-15T04:08:10.768 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 -- start start 2023-11-15T04:08:10.769 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:10.769 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.761+0000 7f4c1897a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c14108d20 con 0x7f4c140faba0 2023-11-15T04:08:10.769 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:10.769 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35166/0 (socket says 172.21.15.38:35166) 2023-11-15T04:08:10.769 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1259c700 1 -- 172.21.15.38:0/692474923 learned_addr learned my addr 172.21.15.38:0/692474923 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:10.770 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1259c700 1 -- 172.21.15.38:0/692474923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0400b200 con 0x7f4c140faba0 2023-11-15T04:08:10.770 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1259c700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4c0400e040 tx=0x7f4c04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:10.770 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0401b410 con 0x7f4c140faba0 2023-11-15T04:08:10.770 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4c0401b9e0 con 0x7f4c140faba0 2023-11-15T04:08:10.770 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c14108f20 con 0x7f4c140faba0 2023-11-15T04:08:10.771 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0401a820 con 0x7f4c140faba0 2023-11-15T04:08:10.772 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.762+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c14109330 con 0x7f4c140faba0 2023-11-15T04:08:10.772 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4c0400f070 con 0x7f4c140faba0 2023-11-15T04:08:10.772 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c14045ad0 con 0x7f4c140faba0 2023-11-15T04:08:10.772 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c02ffd700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 0x7f4bf80344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:10.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4c040412c0 con 0x7f4c140faba0 2023-11-15T04:08:10.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c11d9b700 1 -- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 msgr2=0x7f4bf80344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:10.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.764+0000 7f4c11d9b700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 0x7f4bf80344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:08:10.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.768+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c04019950 con 0x7f4c140faba0 2023-11-15T04:08:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.965+0000 7f4c11d9b700 1 -- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 msgr2=0x7f4bf80344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.965+0000 7f4c11d9b700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 0x7f4bf80344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:08:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.975+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4c1410a3e0 con 0x7f4c140faba0 2023-11-15T04:08:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.975+0000 7f4c02ffd700 1 -- 172.21.15.38:0/692474923 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f4c04016070 con 0x7f4c140faba0 2023-11-15T04:08:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:10.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T04:08:10.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:08:10.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "active_name": "smithi038.bnbogz", 2023-11-15T04:08:10.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:08:10.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 msgr2=0x7f4bf80344b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:08:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 0x7f4bf80344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 msgr2=0x7f4c14108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4c0400e040 tx=0x7f4c04006ee0).stop 2023-11-15T04:08:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 shutdown_connections 2023-11-15T04:08:10.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f4bf8031ff0 0x7f4bf80344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:10.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 --2- 172.21.15.38:0/692474923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c140faba0 0x7f4c14108830 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:10.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.977+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 >> 172.21.15.38:0/692474923 conn(0x7f4c140f63c0 msgr2=0x7f4c140ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:10.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.978+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 shutdown_connections 2023-11-15T04:08:10.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:10.978+0000 7f4c1897a700 1 -- 172.21.15.38:0/692474923 wait complete. 2023-11-15T04:08:11.319 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:10.976295+0000 mon.smithi038 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.38:0/692474923' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:08:11.350 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for the mgr to restart... 2023-11-15T04:08:11.351 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr epoch 5... 2023-11-15T04:08:11.351 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:08:11.740 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b1cf5700 1 Processor -- start 2023-11-15T04:08:11.740 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b1cf5700 1 -- start start 2023-11-15T04:08:11.741 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b1cf5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:11.741 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b1cf5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0fec80 con 0x7f31ac0703d0 2023-11-15T04:08:11.743 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b0cf3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:11.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b0cf3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35170/0 (socket says 172.21.15.38:35170) 2023-11-15T04:08:11.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.739+0000 7f31b0cf3700 1 -- 172.21.15.38:0/2202149938 learned_addr learned my addr 172.21.15.38:0/2202149938 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:11.744 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.740+0000 7f31b0cf3700 1 -- 172.21.15.38:0/2202149938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0fedc0 con 0x7f31ac0703d0 2023-11-15T04:08:11.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.740+0000 7f31b0cf3700 1 --2- 172.21.15.38:0/2202149938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f31a000b5f0 tx=0x7f31a0006ee0).ready entity=mon.0 client_cookie=6ab8f9e181585566 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:11.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.740+0000 7f31ab7fe700 1 -- 172.21.15.38:0/2202149938 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a000d600 con 0x7f31ac0703d0 2023-11-15T04:08:11.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.740+0000 7f31ab7fe700 1 -- 172.21.15.38:0/2202149938 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31a000dbd0 con 0x7f31ac0703d0 2023-11-15T04:08:11.745 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.740+0000 7f31ab7fe700 1 -- 172.21.15.38:0/2202149938 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a0011c00 con 0x7f31ac0703d0 2023-11-15T04:08:11.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 -- 172.21.15.38:0/2202149938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 msgr2=0x7f31ac0fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:11.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/2202149938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f31a000b5f0 tx=0x7f31a0006ee0).stop 2023-11-15T04:08:11.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 -- 172.21.15.38:0/2202149938 shutdown_connections 2023-11-15T04:08:11.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/2202149938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fe790 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:11.746 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 -- 172.21.15.38:0/2202149938 >> 172.21.15.38:0/2202149938 conn(0x7f31ac0f6210 msgr2=0x7f31ac0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 -- 172.21.15.38:0/2202149938 shutdown_connections 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.741+0000 7f31b1cf5700 1 -- 172.21.15.38:0/2202149938 wait complete. 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.742+0000 7f31b1cf5700 1 Processor -- start 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.742+0000 7f31b1cf5700 1 -- start start 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b1cf5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:11.747 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b1cf5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0fa690 con 0x7f31ac0703d0 2023-11-15T04:08:11.748 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b0cf3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:11.748 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b0cf3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35172/0 (socket says 172.21.15.38:35172) 2023-11-15T04:08:11.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b0cf3700 1 -- 172.21.15.38:0/1960763813 learned_addr learned my addr 172.21.15.38:0/1960763813 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:11.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b0cf3700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a000b200 con 0x7f31ac0703d0 2023-11-15T04:08:11.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31b0cf3700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f31a000b190 tx=0x7f31a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:11.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.743+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a001b410 con 0x7f31ac0703d0 2023-11-15T04:08:11.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.744+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f31a001b9e0 con 0x7f31ac0703d0 2023-11-15T04:08:11.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.744+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac0fa830 con 0x7f31ac0703d0 2023-11-15T04:08:11.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.744+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a001a880 con 0x7f31ac0703d0 2023-11-15T04:08:11.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.744+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac0fac40 con 0x7f31ac0703d0 2023-11-15T04:08:11.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f31a000f070 con 0x7f31ac0703d0 2023-11-15T04:08:11.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31a97fa700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:11.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31900328d0 con 0x7f319002fd50 2023-11-15T04:08:11.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f31a00411f0 con 0x7f31ac0703d0 2023-11-15T04:08:11.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31abfff700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 msgr2=0x7f3190032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:11.751 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.745+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:08:11.946 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.946+0000 7f31abfff700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 msgr2=0x7f3190032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:11.947 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:11.946+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:08:12.347 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:12.347+0000 7f31abfff700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 msgr2=0x7f3190032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:12.348 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:12.347+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:08:13.148 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:13.148+0000 7f31abfff700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 msgr2=0x7f3190032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2308732625 2023-11-15T04:08:13.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:13.148+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:08:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:14.246+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f31a001bb50 con 0x7f31ac0703d0 2023-11-15T04:08:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:14.247+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 msgr2=0x7f3190032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:08:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:14.247+0000 7f31a97fa700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:14.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:14.195025+0000 mon.smithi038 (mon.0) 30 : cluster [INF] Active manager daemon smithi038.bnbogz restarted 2023-11-15T04:08:14.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:14.195549+0000 mon.smithi038 (mon.0) 31 : cluster [INF] Activating manager daemon smithi038.bnbogz 2023-11-15T04:08:14.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:14.200726+0000 mon.smithi038 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T04:08:15.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.256+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f31a0019300 con 0x7f31ac0703d0 2023-11-15T04:08:15.257 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.256+0000 7f31a97fa700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 0x7f3190035c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:15.258 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.256+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31900328d0 con 0x7f3190033820 2023-11-15T04:08:15.258 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.257+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 0x7f3190035c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:15.258 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.257+0000 7f31abfff700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 0x7f3190035c10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f31980036d0 tx=0x7f3198006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:15.259 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.258+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f31900328d0 con 0x7f3190033820 2023-11-15T04:08:15.265 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.265+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f31ac059100 con 0x7f3190033820 2023-11-15T04:08:15.266 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.265+0000 7f31a97fa700 1 -- 172.21.15.38:0/1960763813 <== mgr.14116 v2:172.21.15.38:6800/2226868937 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f31ac059100 con 0x7f3190033820 2023-11-15T04:08:15.266 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:15.266 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T04:08:15.267 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:08:15.267 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:15.268 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 msgr2=0x7f3190035c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:15.268 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 0x7f3190035c10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f31980036d0 tx=0x7f3198006ca0).stop 2023-11-15T04:08:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 msgr2=0x7f31ac0fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f31a000b190 tx=0x7f31a0006ee0).stop 2023-11-15T04:08:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 shutdown_connections 2023-11-15T04:08:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f3190033820 0x7f3190035c10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31ac0703d0 0x7f31ac0fc210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 --2- 172.21.15.38:0/1960763813 >> [v2:172.21.15.38:6800/2308732625,v1:172.21.15.38:6801/2308732625] conn(0x7f319002fd50 0x7f3190032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 >> 172.21.15.38:0/1960763813 conn(0x7f31ac0f6210 msgr2=0x7f31ac0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 shutdown_connections 2023-11-15T04:08:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:15.266+0000 7f31b1cf5700 1 -- 172.21.15.38:0/1960763813 wait complete. 2023-11-15T04:08:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:14.249048+0000 mon.smithi038 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi038.bnbogz(active, starting, since 0.0535718s) 2023-11-15T04:08:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.251895+0000 mon.smithi038 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2023-11-15T04:08:15.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.252702+0000 mon.smithi038 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mgr metadata", "who": "smithi038.bnbogz", "id": "smithi038.bnbogz"}]: dispatch 2023-11-15T04:08:15.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.253750+0000 mon.smithi038 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:08:15.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.254018+0000 mon.smithi038 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:08:15.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.254244+0000 mon.smithi038 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:08:15.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:14.269794+0000 mon.smithi038 (mon.0) 39 : cluster [INF] Manager daemon smithi038.bnbogz is now available 2023-11-15T04:08:15.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.286233+0000 mon.smithi038 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:15.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.289577+0000 mon.smithi038 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:15.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.292734+0000 mon.smithi038 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:15.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.398474+0000 mon.smithi038 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:15.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.399292+0000 mon.smithi038 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:15.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.400576+0000 mon.smithi038 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:15.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.406208+0000 mon.smithi038 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:15.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.413765+0000 mon.smithi038 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:08:15.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:14.415123+0000 mon.smithi038 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:08:15.617 INFO:teuthology.orchestra.run.smithi038.stderr:mgr epoch 5 is available 2023-11-15T04:08:15.617 INFO:teuthology.orchestra.run.smithi038.stderr:Setting orchestrator backend to cephadm... 2023-11-15T04:08:15.617 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T04:08:16.582 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:15.258750+0000 mgr.smithi038.bnbogz (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:08:16.583 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:15.259672+0000 mon.smithi038 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi038.bnbogz(active, since 1.06419s) 2023-11-15T04:08:16.583 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:15.266374+0000 mgr.smithi038.bnbogz (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:08:17.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd729c64700 1 Processor -- start 2023-11-15T04:08:17.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd729c64700 1 -- start start 2023-11-15T04:08:17.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd729c64700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:17.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd729c64700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7240fec20 con 0x7fd724070430 2023-11-15T04:08:17.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd728c62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:17.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd728c62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35222/0 (socket says 172.21.15.38:35222) 2023-11-15T04:08:17.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.092+0000 7fd728c62700 1 -- 172.21.15.38:0/1034410984 learned_addr learned my addr 172.21.15.38:0/1034410984 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:17.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.093+0000 7fd728c62700 1 -- 172.21.15.38:0/1034410984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7240fed60 con 0x7fd724070430 2023-11-15T04:08:17.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.093+0000 7fd728c62700 1 --2- 172.21.15.38:0/1034410984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd71800b5f0 tx=0x7fd718006ee0).ready entity=mon.0 client_cookie=4db4c989f967437 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.093+0000 7fd7237fe700 1 -- 172.21.15.38:0/1034410984 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd71800d600 con 0x7fd724070430 2023-11-15T04:08:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.093+0000 7fd7237fe700 1 -- 172.21.15.38:0/1034410984 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd71800dbd0 con 0x7fd724070430 2023-11-15T04:08:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.093+0000 7fd7237fe700 1 -- 172.21.15.38:0/1034410984 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd718011c00 con 0x7fd724070430 2023-11-15T04:08:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- 172.21.15.38:0/1034410984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 msgr2=0x7fd7240fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 --2- 172.21.15.38:0/1034410984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd71800b5f0 tx=0x7fd718006ee0).stop 2023-11-15T04:08:17.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- 172.21.15.38:0/1034410984 shutdown_connections 2023-11-15T04:08:17.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 --2- 172.21.15.38:0/1034410984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd7240fe730 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:17.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- 172.21.15.38:0/1034410984 >> 172.21.15.38:0/1034410984 conn(0x7fd7240f6210 msgr2=0x7fd7240f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:17.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- 172.21.15.38:0/1034410984 shutdown_connections 2023-11-15T04:08:17.156 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- 172.21.15.38:0/1034410984 wait complete. 2023-11-15T04:08:17.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 Processor -- start 2023-11-15T04:08:17.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.094+0000 7fd729c64700 1 -- start start 2023-11-15T04:08:17.157 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd729c64700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:17.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd729c64700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72410ad20 con 0x7fd724070430 2023-11-15T04:08:17.158 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd728c62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:17.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd728c62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35224/0 (socket says 172.21.15.38:35224) 2023-11-15T04:08:17.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd728c62700 1 -- 172.21.15.38:0/2296464797 learned_addr learned my addr 172.21.15.38:0/2296464797 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:17.159 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd728c62700 1 -- 172.21.15.38:0/2296464797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71800b200 con 0x7fd724070430 2023-11-15T04:08:17.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd728c62700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd71800b190 tx=0x7fd718006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:17.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd71801b410 con 0x7fd724070430 2023-11-15T04:08:17.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72410af20 con 0x7fd724070430 2023-11-15T04:08:17.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd71801b9e0 con 0x7fd724070430 2023-11-15T04:08:17.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd71801a880 con 0x7fd724070430 2023-11-15T04:08:17.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.095+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72410b330 con 0x7fd724070430 2023-11-15T04:08:17.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.096+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd71800f070 con 0x7fd724070430 2023-11-15T04:08:17.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.096+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd724045ad0 con 0x7fd724070430 2023-11-15T04:08:17.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.097+0000 7fd7217fa700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 0x7fd708032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:17.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.097+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd7180412b0 con 0x7fd724070430 2023-11-15T04:08:17.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.097+0000 7fd723fff700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 0x7fd708032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:17.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.097+0000 7fd723fff700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 0x7fd708032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7100096a0 tx=0x7fd710006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:17.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.101+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd718022310 con 0x7fd724070430 2023-11-15T04:08:17.305 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.304+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd72410bc70 con 0x7fd70802fd50 2023-11-15T04:08:17.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.310+0000 7fd7217fa700 1 -- 172.21.15.38:0/2296464797 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd72410bc70 con 0x7fd70802fd50 2023-11-15T04:08:17.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 msgr2=0x7fd708032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:17.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 0x7fd708032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7100096a0 tx=0x7fd710006b40).stop 2023-11-15T04:08:17.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 msgr2=0x7fd72410a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:17.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd71800b190 tx=0x7fd718006ee0).stop 2023-11-15T04:08:17.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 shutdown_connections 2023-11-15T04:08:17.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd70802fd50 0x7fd708032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:17.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 --2- 172.21.15.38:0/2296464797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd724070430 0x7fd72410a830 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:17.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 >> 172.21.15.38:0/2296464797 conn(0x7fd7240f6210 msgr2=0x7fd7240f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:17.316 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 shutdown_connections 2023-11-15T04:08:17.316 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:17.312+0000 7fd729c64700 1 -- 172.21.15.38:0/2296464797 wait complete. 2023-11-15T04:08:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:16.264274+0000 mon.smithi038 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi038.bnbogz(active, since 2s) 2023-11-15T04:08:18.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:18 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:17.306287+0000 mgr.smithi038.bnbogz (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:18.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:18 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:17.310664+0000 mon.smithi038 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:18.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:18 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:17.314775+0000 mon.smithi038 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:19.313 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T04:08:20.800 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.689+0000 7f89c76ae700 1 Processor -- start 2023-11-15T04:08:20.800 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.689+0000 7f89c76ae700 1 -- start start 2023-11-15T04:08:20.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.689+0000 7f89c76ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:20.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.689+0000 7f89c76ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c00fc730 con 0x7f89c00703d0 2023-11-15T04:08:20.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89c544a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:20.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89c544a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46932/0 (socket says 172.21.15.38:46932) 2023-11-15T04:08:20.801 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89c544a700 1 -- 172.21.15.38:0/2104963773 learned_addr learned my addr 172.21.15.38:0/2104963773 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:20.802 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89c544a700 1 -- 172.21.15.38:0/2104963773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0101230 con 0x7f89c00703d0 2023-11-15T04:08:20.802 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89c544a700 1 --2- 172.21.15.38:0/2104963773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f89b400b5f0 tx=0x7f89b4006ee0).ready entity=mon.0 client_cookie=36b5ccd5d85f7347 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:20.802 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89b3fff700 1 -- 172.21.15.38:0/2104963773 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b400d600 con 0x7f89c00703d0 2023-11-15T04:08:20.802 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89b3fff700 1 -- 172.21.15.38:0/2104963773 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89b400dbd0 con 0x7f89c00703d0 2023-11-15T04:08:20.802 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.690+0000 7f89b3fff700 1 -- 172.21.15.38:0/2104963773 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b4011c00 con 0x7f89c00703d0 2023-11-15T04:08:20.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 -- 172.21.15.38:0/2104963773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 msgr2=0x7f89c0100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:20.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2104963773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f89b400b5f0 tx=0x7f89b4006ee0).stop 2023-11-15T04:08:20.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 -- 172.21.15.38:0/2104963773 shutdown_connections 2023-11-15T04:08:20.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2104963773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c0100d40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:20.803 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 -- 172.21.15.38:0/2104963773 >> 172.21.15.38:0/2104963773 conn(0x7f89c00f63c0 msgr2=0x7f89c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:20.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 -- 172.21.15.38:0/2104963773 shutdown_connections 2023-11-15T04:08:20.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.691+0000 7f89c76ae700 1 -- 172.21.15.38:0/2104963773 wait complete. 2023-11-15T04:08:20.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c76ae700 1 Processor -- start 2023-11-15T04:08:20.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c76ae700 1 -- start start 2023-11-15T04:08:20.804 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c76ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:20.805 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c76ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c010d0a0 con 0x7f89c00703d0 2023-11-15T04:08:20.805 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c544a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:20.805 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c544a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46944/0 (socket says 172.21.15.38:46944) 2023-11-15T04:08:20.805 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.692+0000 7f89c544a700 1 -- 172.21.15.38:0/2948292873 learned_addr learned my addr 172.21.15.38:0/2948292873 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:20.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89c544a700 1 -- 172.21.15.38:0/2948292873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b400b200 con 0x7f89c00703d0 2023-11-15T04:08:20.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89c544a700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f89b400b190 tx=0x7f89b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:20.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b401b410 con 0x7f89c00703d0 2023-11-15T04:08:20.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89b401b9e0 con 0x7f89c00703d0 2023-11-15T04:08:20.806 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c010d2a0 con 0x7f89c00703d0 2023-11-15T04:08:20.807 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b401a880 con 0x7f89c00703d0 2023-11-15T04:08:20.807 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.693+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c010d6b0 con 0x7f89c00703d0 2023-11-15T04:08:20.807 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.694+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f89b400f070 con 0x7f89c00703d0 2023-11-15T04:08:20.807 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.694+0000 7f89b1ffb700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 0x7f89a8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:20.807 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.694+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f89b4041330 con 0x7f89c00703d0 2023-11-15T04:08:20.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.694+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c0045ad0 con 0x7f89c00703d0 2023-11-15T04:08:20.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.694+0000 7f89c4c49700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 0x7f89a8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:20.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.695+0000 7f89c4c49700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 0x7f89a8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f89bc0096a0 tx=0x7f89bc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:20.808 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.699+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89b4016080 con 0x7f89c00703d0 2023-11-15T04:08:20.869 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.868+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f89c0106b70 con 0x7f89a802fd50 2023-11-15T04:08:20.870 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.868+0000 7f89b1ffb700 1 -- 172.21.15.38:0/2948292873 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f89c0106b70 con 0x7f89a802fd50 2023-11-15T04:08:20.870 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: value unchanged 2023-11-15T04:08:20.872 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 msgr2=0x7f89a8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:20.873 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 0x7f89a8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f89bc0096a0 tx=0x7f89bc006b40).stop 2023-11-15T04:08:20.873 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 msgr2=0x7f89c010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:20.873 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f89b400b190 tx=0x7f89b4006ee0).stop 2023-11-15T04:08:20.873 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 shutdown_connections 2023-11-15T04:08:20.873 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f89a802fd50 0x7f89a8032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:20.874 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 --2- 172.21.15.38:0/2948292873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89c00703d0 0x7f89c010cbb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:20.874 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 >> 172.21.15.38:0/2948292873 conn(0x7f89c00f63c0 msgr2=0x7f89c00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:20.874 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 shutdown_connections 2023-11-15T04:08:20.874 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:20.870+0000 7f89c76ae700 1 -- 172.21.15.38:0/2948292873 wait complete. 2023-11-15T04:08:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:20.870547+0000 mgr.smithi038.bnbogz (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:23.065 INFO:teuthology.orchestra.run.smithi038.stderr:Generating ssh key... 2023-11-15T04:08:23.066 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T04:08:23.820 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.818+0000 7f486359e700 1 Processor -- start 2023-11-15T04:08:23.821 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.818+0000 7f486359e700 1 -- start start 2023-11-15T04:08:23.821 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:23.821 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486359e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4864070770 con 0x7f4864070350 2023-11-15T04:08:23.826 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:23.826 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46948/0 (socket says 172.21.15.38:46948) 2023-11-15T04:08:23.826 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486259c700 1 -- 172.21.15.38:0/3601969570 learned_addr learned my addr 172.21.15.38:0/3601969570 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:23.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486259c700 1 -- 172.21.15.38:0/3601969570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48640fed00 con 0x7f4864070350 2023-11-15T04:08:23.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.819+0000 7f486259c700 1 --2- 172.21.15.38:0/3601969570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f485400b5f0 tx=0x7f4854006ee0).ready entity=mon.0 client_cookie=aedb4ffc4556150a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:23.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486159a700 1 -- 172.21.15.38:0/3601969570 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f485400d600 con 0x7f4864070350 2023-11-15T04:08:23.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486159a700 1 -- 172.21.15.38:0/3601969570 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f485400dbd0 con 0x7f4864070350 2023-11-15T04:08:23.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486159a700 1 -- 172.21.15.38:0/3601969570 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4854011c00 con 0x7f4864070350 2023-11-15T04:08:23.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486359e700 1 -- 172.21.15.38:0/3601969570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 msgr2=0x7f48640fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:23.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486359e700 1 --2- 172.21.15.38:0/3601969570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f485400b5f0 tx=0x7f4854006ee0).stop 2023-11-15T04:08:23.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486359e700 1 -- 172.21.15.38:0/3601969570 shutdown_connections 2023-11-15T04:08:23.829 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486359e700 1 --2- 172.21.15.38:0/3601969570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f48640fe810 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:23.829 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.820+0000 7f486359e700 1 -- 172.21.15.38:0/3601969570 >> 172.21.15.38:0/3601969570 conn(0x7f48640f6210 msgr2=0x7f48640f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:23.829 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 -- 172.21.15.38:0/3601969570 shutdown_connections 2023-11-15T04:08:23.830 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 -- 172.21.15.38:0/3601969570 wait complete. 2023-11-15T04:08:23.830 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 Processor -- start 2023-11-15T04:08:23.830 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 -- start start 2023-11-15T04:08:23.830 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:23.831 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.821+0000 7f486359e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486410acf0 con 0x7f4864070350 2023-11-15T04:08:23.831 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:23.831 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46954/0 (socket says 172.21.15.38:46954) 2023-11-15T04:08:23.831 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486259c700 1 -- 172.21.15.38:0/402262421 learned_addr learned my addr 172.21.15.38:0/402262421 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:23.832 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486259c700 1 -- 172.21.15.38:0/402262421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485400b200 con 0x7f4864070350 2023-11-15T04:08:23.832 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486259c700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f485400e040 tx=0x7f4854006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:23.832 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f485400d9f0 con 0x7f4864070350 2023-11-15T04:08:23.833 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f485401a9f0 con 0x7f4864070350 2023-11-15T04:08:23.833 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f485401b890 con 0x7f4864070350 2023-11-15T04:08:23.833 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486410aef0 con 0x7f4864070350 2023-11-15T04:08:23.834 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.822+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486410b300 con 0x7f4864070350 2023-11-15T04:08:23.834 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.823+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f485400f070 con 0x7f4864070350 2023-11-15T04:08:23.834 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.823+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4864045ad0 con 0x7f4864070350 2023-11-15T04:08:23.834 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.823+0000 7f4852ffd700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 0x7f4848032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:23.835 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.823+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f48540412c0 con 0x7f4864070350 2023-11-15T04:08:23.835 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.824+0000 7f4861d9b700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 0x7f4848032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:23.835 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.824+0000 7f4861d9b700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 0x7f4848032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48580096a0 tx=0x7f4858006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:23.836 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.828+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48540224f0 con 0x7f4864070350 2023-11-15T04:08:23.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:23.994+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f486410bbe0 con 0x7f484802fdb0 2023-11-15T04:08:24.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.295+0000 7f4852ffd700 1 -- 172.21.15.38:0/402262421 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f486410bbe0 con 0x7f484802fdb0 2023-11-15T04:08:24.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 msgr2=0x7f4848032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:24.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 0x7f4848032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48580096a0 tx=0x7f4858006b40).stop 2023-11-15T04:08:24.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 msgr2=0x7f486410a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:24.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f485400e040 tx=0x7f4854006ee0).stop 2023-11-15T04:08:24.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 shutdown_connections 2023-11-15T04:08:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f484802fdb0 0x7f4848032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 --2- 172.21.15.38:0/402262421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4864070350 0x7f486410a800 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.297+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 >> 172.21.15.38:0/402262421 conn(0x7f48640f6210 msgr2=0x7f48640f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.298+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 shutdown_connections 2023-11-15T04:08:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:24.298+0000 7f486359e700 1 -- 172.21.15.38:0/402262421 wait complete. 2023-11-15T04:08:24.699 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T04:08:25.069 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.066+0000 7f2d5cc2f700 1 Processor -- start 2023-11-15T04:08:25.069 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.066+0000 7f2d5cc2f700 1 -- start start 2023-11-15T04:08:25.069 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5cc2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.069 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5cc2f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580fee10 con 0x7f2d580703f0 2023-11-15T04:08:25.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46964/0 (socket says 172.21.15.38:46964) 2023-11-15T04:08:25.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5659c700 1 -- 172.21.15.38:0/377901992 learned_addr learned my addr 172.21.15.38:0/377901992 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:25.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5659c700 1 -- 172.21.15.38:0/377901992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580fef50 con 0x7f2d580703f0 2023-11-15T04:08:25.070 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.067+0000 7f2d5659c700 1 --2- 172.21.15.38:0/377901992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2d4800b5f0 tx=0x7f2d48006ee0).ready entity=mon.0 client_cookie=62bfe11bbba9d1fc server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d55d9b700 1 -- 172.21.15.38:0/377901992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4800d600 con 0x7f2d580703f0 2023-11-15T04:08:25.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d55d9b700 1 -- 172.21.15.38:0/377901992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d4800dbd0 con 0x7f2d580703f0 2023-11-15T04:08:25.071 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d55d9b700 1 -- 172.21.15.38:0/377901992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d48011c00 con 0x7f2d580703f0 2023-11-15T04:08:25.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/377901992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 msgr2=0x7f2d580fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:25.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/377901992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2d4800b5f0 tx=0x7f2d48006ee0).stop 2023-11-15T04:08:25.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/377901992 shutdown_connections 2023-11-15T04:08:25.074 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/377901992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fe920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:25.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/377901992 >> 172.21.15.38:0/377901992 conn(0x7f2d580f63c0 msgr2=0x7f2d580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:25.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/377901992 shutdown_connections 2023-11-15T04:08:25.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.068+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/377901992 wait complete. 2023-11-15T04:08:25.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.069+0000 7f2d5cc2f700 1 Processor -- start 2023-11-15T04:08:25.075 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.069+0000 7f2d5cc2f700 1 -- start start 2023-11-15T04:08:25.076 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.069+0000 7f2d5cc2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.076 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.069+0000 7f2d5cc2f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580fa820 con 0x7f2d580703f0 2023-11-15T04:08:25.076 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.076 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46972/0 (socket says 172.21.15.38:46972) 2023-11-15T04:08:25.076 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5659c700 1 -- 172.21.15.38:0/1289425808 learned_addr learned my addr 172.21.15.38:0/1289425808 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:25.077 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5659c700 1 -- 172.21.15.38:0/1289425808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4800b200 con 0x7f2d580703f0 2023-11-15T04:08:25.077 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5659c700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d4800b190 tx=0x7f2d48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.077 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4801b410 con 0x7f2d580703f0 2023-11-15T04:08:25.077 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d4801b9e0 con 0x7f2d580703f0 2023-11-15T04:08:25.077 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4801a880 con 0x7f2d580703f0 2023-11-15T04:08:25.078 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d580fa9c0 con 0x7f2d580703f0 2023-11-15T04:08:25.078 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.070+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d580fadd0 con 0x7f2d580703f0 2023-11-15T04:08:25.078 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.071+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2d4800f070 con 0x7f2d580703f0 2023-11-15T04:08:25.078 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.071+0000 7f2d4f7fe700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 0x7f2d38032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.078 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.071+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d58045ad0 con 0x7f2d580703f0 2023-11-15T04:08:25.079 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.071+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2d480412a0 con 0x7f2d580703f0 2023-11-15T04:08:25.079 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.071+0000 7f2d4dd9b700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 0x7f2d38032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.079 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.072+0000 7f2d4dd9b700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 0x7f2d38032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2d400096a0 tx=0x7f2d40006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.080 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.076+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d48016070 con 0x7f2d580703f0 2023-11-15T04:08:25.243 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.243+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d580fb010 con 0x7f2d3802fdb0 2023-11-15T04:08:25.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.243+0000 7f2d4f7fe700 1 -- 172.21.15.38:0/1289425808 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2d580fb010 con 0x7f2d3802fdb0 2023-11-15T04:08:25.244 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAPzcLBh8oQd+ZPKGSU3Yqs8C4Iv0wSo2Z9jWPNZmoau26qVWs9/8LlVeCIHNaMtyGVEJokXXv+6VtGYU7Te9LiW//mO5pqFD2PErjjERHWggbUDSJ0z6ty8y/xPoEgmy65FPZ7JNwLX4wgdez7eOkbU2Z008MPdNktY4iOs793Vvkx1qNVQi8tWwI3Rqcz/r5GyU4qrQ2P+XzjvtKlDaY6ZN63GkFF/rviS9EWIBDJcZcSTB9jrOnua+ufU2Ootx/BA3aD9Usze5hE7bwoZnatLknFVp0wdJ3vwkEL5Cwho4M3CltPMuNJEANmTh54mhbdqLmLQLU52rEe1zrSA3In8rLN4TpuGT3KKtWAYTze7lQtQ/zLE0sh1pq+cLIA1eY/4WgPv6RGfOfi5EonoxJzt7+mem5rWVNzt0YJvAegzXobca2Znyoc8qhjgBCV87uVIcyMH8X77fxQhEWp2qEOLfp3JX/Uyt6lj0oV0THbMLBIHW38gkzAxvz2jcqUU0= ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:08:25.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 msgr2=0x7f2d38032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:25.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 0x7f2d38032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2d400096a0 tx=0x7f2d40006b40).stop 2023-11-15T04:08:25.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 msgr2=0x7f2d580fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:25.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d4800b190 tx=0x7f2d48006ee0).stop 2023-11-15T04:08:25.247 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 shutdown_connections 2023-11-15T04:08:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f2d3802fdb0 0x7f2d38032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 --2- 172.21.15.38:0/1289425808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d580703f0 0x7f2d580fc3a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 >> 172.21.15.38:0/1289425808 conn(0x7f2d580f63c0 msgr2=0x7f2d580f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 shutdown_connections 2023-11-15T04:08:25.248 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.245+0000 7f2d5cc2f700 1 -- 172.21.15.38:0/1289425808 wait complete. 2023-11-15T04:08:25.559 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:23.997611+0000 mgr.smithi038.bnbogz (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-15T04:08:25.560 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:24.294472+0000 mon.smithi038 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:25.560 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:24.296687+0000 mon.smithi038 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:08:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T04:08:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:Adding host smithi038... 2023-11-15T04:08:25.592 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi038 172.21.15.38 2023-11-15T04:08:25.971 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.969+0000 7fa5a60c0700 1 Processor -- start 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.969+0000 7fa5a60c0700 1 -- start start 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa5a60c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa5a60c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a0070770 con 0x7fa5a0070350 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa59f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa59f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46982/0 (socket says 172.21.15.38:46982) 2023-11-15T04:08:25.972 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa59f7fe700 1 -- 172.21.15.38:0/2995289529 learned_addr learned my addr 172.21.15.38:0/2995289529 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:25.973 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa59f7fe700 1 -- 172.21.15.38:0/2995289529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00feeb0 con 0x7fa5a0070350 2023-11-15T04:08:25.973 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.970+0000 7fa59f7fe700 1 --2- 172.21.15.38:0/2995289529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa59000b5f0 tx=0x7fa590006ee0).ready entity=mon.0 client_cookie=d8fa552b51f55021 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.973 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa59e7fc700 1 -- 172.21.15.38:0/2995289529 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa59000d600 con 0x7fa5a0070350 2023-11-15T04:08:25.973 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa59e7fc700 1 -- 172.21.15.38:0/2995289529 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa59000dbd0 con 0x7fa5a0070350 2023-11-15T04:08:25.976 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa59e7fc700 1 -- 172.21.15.38:0/2995289529 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa590011c00 con 0x7fa5a0070350 2023-11-15T04:08:25.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2995289529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 msgr2=0x7fa5a00fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:25.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2995289529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa59000b5f0 tx=0x7fa590006ee0).stop 2023-11-15T04:08:25.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2995289529 shutdown_connections 2023-11-15T04:08:25.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2995289529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a00fe9c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:25.977 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2995289529 >> 172.21.15.38:0/2995289529 conn(0x7fa5a00f63c0 msgr2=0x7fa5a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2995289529 shutdown_connections 2023-11-15T04:08:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.971+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2995289529 wait complete. 2023-11-15T04:08:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.972+0000 7fa5a60c0700 1 Processor -- start 2023-11-15T04:08:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.972+0000 7fa5a60c0700 1 -- start start 2023-11-15T04:08:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa5a60c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa5a60c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a010af40 con 0x7fa5a0070350 2023-11-15T04:08:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa59f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa59f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46984/0 (socket says 172.21.15.38:46984) 2023-11-15T04:08:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa59f7fe700 1 -- 172.21.15.38:0/2825649926 learned_addr learned my addr 172.21.15.38:0/2825649926 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa59f7fe700 1 -- 172.21.15.38:0/2825649926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59000b200 con 0x7fa5a0070350 2023-11-15T04:08:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.973+0000 7fa59f7fe700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa590002f40 tx=0x7fa590006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa590011690 con 0x7fa5a0070350 2023-11-15T04:08:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa59001b880 con 0x7fa5a0070350 2023-11-15T04:08:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a010b140 con 0x7fa5a0070350 2023-11-15T04:08:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa59001a770 con 0x7fa5a0070350 2023-11-15T04:08:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a010b550 con 0x7fa5a0070350 2023-11-15T04:08:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.974+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa59000f070 con 0x7fa5a0070350 2023-11-15T04:08:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.975+0000 7fa587fff700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 0x7fa58c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.975+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a0045ad0 con 0x7fa5a0070350 2023-11-15T04:08:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.975+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa5900411d0 con 0x7fa5a0070350 2023-11-15T04:08:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.976+0000 7fa59effd700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 0x7fa58c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.976+0000 7fa59effd700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 0x7fa58c032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa5940096a0 tx=0x7fa594006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:25.980+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa59006a2b0 con 0x7fa5a0070350 2023-11-15T04:08:26.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.169+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5a010be30 con 0x7fa58c02fd50 2023-11-15T04:08:26.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:25.244538+0000 mgr.smithi038.bnbogz (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:26.824 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Added host 'smithi038' 2023-11-15T04:08:26.824 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.822+0000 7fa587fff700 1 -- 172.21.15.38:0/2825649926 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fa5a010be30 con 0x7fa58c02fd50 2023-11-15T04:08:26.825 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 msgr2=0x7fa58c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:26.826 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 0x7fa58c032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa5940096a0 tx=0x7fa594006b40).stop 2023-11-15T04:08:26.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 msgr2=0x7fa5a010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:26.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa590002f40 tx=0x7fa590006ee0).stop 2023-11-15T04:08:26.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 shutdown_connections 2023-11-15T04:08:26.827 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fa58c02fd50 0x7fa58c032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:26.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 --2- 172.21.15.38:0/2825649926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5a0070350 0x7fa5a010aa50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:26.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 >> 172.21.15.38:0/2825649926 conn(0x7fa5a00f63c0 msgr2=0x7fa5a00f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:26.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 shutdown_connections 2023-11-15T04:08:26.828 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:26.825+0000 7fa5a60c0700 1 -- 172.21.15.38:0/2825649926 wait complete. 2023-11-15T04:08:27.134 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying mon service with default placement... 2023-11-15T04:08:27.134 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T04:08:27.494 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.492+0000 7fc4d0086700 1 Processor -- start 2023-11-15T04:08:27.498 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4d0086700 1 -- start start 2023-11-15T04:08:27.498 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4d0086700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:27.498 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4d0086700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80fc750 con 0x7fc4c80703b0 2023-11-15T04:08:27.499 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4cde22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:27.499 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4cde22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57900/0 (socket says 172.21.15.38:57900) 2023-11-15T04:08:27.499 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.493+0000 7fc4cde22700 1 -- 172.21.15.38:0/1180005197 learned_addr learned my addr 172.21.15.38:0/1180005197 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:27.499 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.494+0000 7fc4cde22700 1 -- 172.21.15.38:0/1180005197 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8101250 con 0x7fc4c80703b0 2023-11-15T04:08:27.499 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.494+0000 7fc4cde22700 1 --2- 172.21.15.38:0/1180005197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc4c400b5f0 tx=0x7fc4c4006ee0).ready entity=mon.0 client_cookie=ec253e5ad1b05bab server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:27.500 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.494+0000 7fc4cce20700 1 -- 172.21.15.38:0/1180005197 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c400d600 con 0x7fc4c80703b0 2023-11-15T04:08:27.500 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.494+0000 7fc4cce20700 1 -- 172.21.15.38:0/1180005197 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc4c400dbd0 con 0x7fc4c80703b0 2023-11-15T04:08:27.500 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.494+0000 7fc4cce20700 1 -- 172.21.15.38:0/1180005197 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c4011c00 con 0x7fc4c80703b0 2023-11-15T04:08:27.500 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- 172.21.15.38:0/1180005197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 msgr2=0x7fc4c8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:27.500 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 --2- 172.21.15.38:0/1180005197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc4c400b5f0 tx=0x7fc4c4006ee0).stop 2023-11-15T04:08:27.501 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- 172.21.15.38:0/1180005197 shutdown_connections 2023-11-15T04:08:27.501 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 --2- 172.21.15.38:0/1180005197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c8100d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:27.501 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- 172.21.15.38:0/1180005197 >> 172.21.15.38:0/1180005197 conn(0x7fc4c80f63c0 msgr2=0x7fc4c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:27.501 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- 172.21.15.38:0/1180005197 shutdown_connections 2023-11-15T04:08:27.501 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- 172.21.15.38:0/1180005197 wait complete. 2023-11-15T04:08:27.502 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 Processor -- start 2023-11-15T04:08:27.502 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.495+0000 7fc4d0086700 1 -- start start 2023-11-15T04:08:27.502 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4d0086700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:27.502 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4d0086700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c810d1e0 con 0x7fc4c80703b0 2023-11-15T04:08:27.502 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4cde22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:27.503 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4cde22700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57912/0 (socket says 172.21.15.38:57912) 2023-11-15T04:08:27.503 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4cde22700 1 -- 172.21.15.38:0/3826303477 learned_addr learned my addr 172.21.15.38:0/3826303477 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:27.503 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4cde22700 1 -- 172.21.15.38:0/3826303477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c400b200 con 0x7fc4c80703b0 2023-11-15T04:08:27.503 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4cde22700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc4c400e040 tx=0x7fc4c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:27.503 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c400d9f0 con 0x7fc4c80703b0 2023-11-15T04:08:27.504 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.496+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc4c401a9f0 con 0x7fc4c80703b0 2023-11-15T04:08:27.504 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.497+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c810d3e0 con 0x7fc4c80703b0 2023-11-15T04:08:27.504 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.497+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c401b8c0 con 0x7fc4c80703b0 2023-11-15T04:08:27.504 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.497+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c810d7f0 con 0x7fc4c80703b0 2023-11-15T04:08:27.504 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.497+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc4c400f070 con 0x7fc4c80703b0 2023-11-15T04:08:27.505 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.497+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c8045ad0 con 0x7fc4c80703b0 2023-11-15T04:08:27.505 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.498+0000 7fc4b67fc700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 0x7fc4b00322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:27.506 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.498+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc4c4041390 con 0x7fc4c80703b0 2023-11-15T04:08:27.506 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.498+0000 7fc4cd621700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 0x7fc4b00322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:27.506 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.498+0000 7fc4cd621700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 0x7fc4b00322c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc4b80096a0 tx=0x7fc4b8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:27.506 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.502+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4c4022d20 con 0x7fc4c80703b0 2023-11-15T04:08:27.696 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.696+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4c810e050 con 0x7fc4b002fe00 2023-11-15T04:08:27.701 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.700+0000 7fc4b67fc700 1 -- 172.21.15.38:0/3826303477 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc4c810e050 con 0x7fc4b002fe00 2023-11-15T04:08:27.701 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T04:08:27.703 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 msgr2=0x7fc4b00322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:27.703 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 0x7fc4b00322c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc4b80096a0 tx=0x7fc4b8006b40).stop 2023-11-15T04:08:27.703 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 msgr2=0x7fc4c810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:27.703 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc4c400e040 tx=0x7fc4c4006ee0).stop 2023-11-15T04:08:27.704 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 shutdown_connections 2023-11-15T04:08:27.704 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fc4b002fe00 0x7fc4b00322c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 --2- 172.21.15.38:0/3826303477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c80703b0 0x7fc4c810ccf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 >> 172.21.15.38:0/3826303477 conn(0x7fc4c80f63c0 msgr2=0x7fc4c80f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 shutdown_connections 2023-11-15T04:08:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:27.702+0000 7fc4d0086700 1 -- 172.21.15.38:0/3826303477 wait complete. 2023-11-15T04:08:27.723 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:26.170790+0000 mgr.smithi038.bnbogz (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:27.724 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:26.822791+0000 mon.smithi038 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:27.724 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:26.823425+0000 mon.smithi038 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:28.040 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying mgr service with default placement... 2023-11-15T04:08:28.041 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T04:08:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.421+0000 7f0c81c16700 1 Processor -- start 2023-11-15T04:08:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.421+0000 7f0c81c16700 1 -- start start 2023-11-15T04:08:28.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c81c16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:28.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c81c16700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c0fee10 con 0x7f0c7c0703f0 2023-11-15T04:08:28.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:28.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57922/0 (socket says 172.21.15.38:57922) 2023-11-15T04:08:28.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7b7fe700 1 -- 172.21.15.38:0/3311191255 learned_addr learned my addr 172.21.15.38:0/3311191255 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:28.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7b7fe700 1 -- 172.21.15.38:0/3311191255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c0fef50 con 0x7f0c7c0703f0 2023-11-15T04:08:28.428 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7b7fe700 1 --2- 172.21.15.38:0/3311191255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0c6c00b5f0 tx=0x7f0c6c006ee0).ready entity=mon.0 client_cookie=22744946f71d2a2a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:28.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.422+0000 7f0c7a7fc700 1 -- 172.21.15.38:0/3311191255 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c6c00d600 con 0x7f0c7c0703f0 2023-11-15T04:08:28.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c7a7fc700 1 -- 172.21.15.38:0/3311191255 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c6c00dbd0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c7a7fc700 1 -- 172.21.15.38:0/3311191255 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c6c011c00 con 0x7f0c7c0703f0 2023-11-15T04:08:28.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 -- 172.21.15.38:0/3311191255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 msgr2=0x7f0c7c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:28.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 --2- 172.21.15.38:0/3311191255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0c6c00b5f0 tx=0x7f0c6c006ee0).stop 2023-11-15T04:08:28.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 -- 172.21.15.38:0/3311191255 shutdown_connections 2023-11-15T04:08:28.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 --2- 172.21.15.38:0/3311191255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c0fe920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:28.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 -- 172.21.15.38:0/3311191255 >> 172.21.15.38:0/3311191255 conn(0x7f0c7c0f63c0 msgr2=0x7f0c7c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:28.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 -- 172.21.15.38:0/3311191255 shutdown_connections 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.423+0000 7f0c81c16700 1 -- 172.21.15.38:0/3311191255 wait complete. 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.424+0000 7f0c81c16700 1 Processor -- start 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.424+0000 7f0c81c16700 1 -- start start 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.424+0000 7f0c81c16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.424+0000 7f0c81c16700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c10aeb0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:28.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57924/0 (socket says 172.21.15.38:57924) 2023-11-15T04:08:28.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c7b7fe700 1 -- 172.21.15.38:0/4018761906 learned_addr learned my addr 172.21.15.38:0/4018761906 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:28.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c7b7fe700 1 -- 172.21.15.38:0/4018761906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c00b200 con 0x7f0c7c0703f0 2023-11-15T04:08:28.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c7b7fe700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0c6c00b190 tx=0x7f0c6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:28.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c6c01b410 con 0x7f0c7c0703f0 2023-11-15T04:08:28.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c10b0b0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c6c01b9e0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c6c01a880 con 0x7f0c7c0703f0 2023-11-15T04:08:28.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.425+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c10b4c0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.426+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c045ad0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.426+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0c6c00f070 con 0x7f0c7c0703f0 2023-11-15T04:08:28.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.427+0000 7f0c63fff700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 0x7f0c68032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:28.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.427+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0c6c0412b0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.427+0000 7f0c7affd700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 0x7f0c68032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:28.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.427+0000 7f0c7affd700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 0x7f0c68032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0c700096a0 tx=0x7f0c70006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:28.435 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.431+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c6c0224f0 con 0x7f0c7c0703f0 2023-11-15T04:08:28.626 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.626+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c7c10be00 con 0x7f0c6802fd50 2023-11-15T04:08:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.630+0000 7f0c63fff700 1 -- 172.21.15.38:0/4018761906 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0c7c10be00 con 0x7f0c6802fd50 2023-11-15T04:08:28.630 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T04:08:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.631+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 msgr2=0x7f0c68032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.631+0000 7f0c81c16700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 0x7f0c68032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0c700096a0 tx=0x7f0c70006b40).stop 2023-11-15T04:08:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.631+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 msgr2=0x7f0c7c10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.631+0000 7f0c81c16700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0c6c00b190 tx=0x7f0c6c006ee0).stop 2023-11-15T04:08:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 shutdown_connections 2023-11-15T04:08:28.634 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f0c6802fd50 0x7f0c68032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:28.634 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 --2- 172.21.15.38:0/4018761906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c7c0703f0 0x7f0c7c10a9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:28.634 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 >> 172.21.15.38:0/4018761906 conn(0x7f0c7c0f63c0 msgr2=0x7f0c7c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:28.634 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 shutdown_connections 2023-11-15T04:08:28.634 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:28.632+0000 7f0c81c16700 1 -- 172.21.15.38:0/4018761906 wait complete. 2023-11-15T04:08:28.660 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:26.823159+0000 mgr.smithi038.bnbogz (mgr.14116) 9 : cephadm [INF] Added host smithi038 2023-11-15T04:08:28.660 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:27.701103+0000 mon.smithi038 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying crash service with default placement... 2023-11-15T04:08:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T04:08:29.420 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68fab52700 1 Processor -- start 2023-11-15T04:08:29.420 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68fab52700 1 -- start start 2023-11-15T04:08:29.420 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68fab52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:29.421 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68fab52700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40fede0 con 0x7f68f40704a0 2023-11-15T04:08:29.421 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68f88ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:29.423 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68f88ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57926/0 (socket says 172.21.15.38:57926) 2023-11-15T04:08:29.423 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68f88ee700 1 -- 172.21.15.38:0/3342129487 learned_addr learned my addr 172.21.15.38:0/3342129487 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:29.423 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.419+0000 7f68f88ee700 1 -- 172.21.15.38:0/3342129487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40fef20 con 0x7f68f40704a0 2023-11-15T04:08:29.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.420+0000 7f68f88ee700 1 --2- 172.21.15.38:0/3342129487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f68e800b5f0 tx=0x7f68e8006ee0).ready entity=mon.0 client_cookie=4fd43044a80a59c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:29.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.420+0000 7f68f37fe700 1 -- 172.21.15.38:0/3342129487 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e800d600 con 0x7f68f40704a0 2023-11-15T04:08:29.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.420+0000 7f68f37fe700 1 -- 172.21.15.38:0/3342129487 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f68e800dbd0 con 0x7f68f40704a0 2023-11-15T04:08:29.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.420+0000 7f68f37fe700 1 -- 172.21.15.38:0/3342129487 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e8011c00 con 0x7f68f40704a0 2023-11-15T04:08:29.424 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 -- 172.21.15.38:0/3342129487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 msgr2=0x7f68f40fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:29.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 --2- 172.21.15.38:0/3342129487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f68e800b5f0 tx=0x7f68e8006ee0).stop 2023-11-15T04:08:29.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 -- 172.21.15.38:0/3342129487 shutdown_connections 2023-11-15T04:08:29.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 --2- 172.21.15.38:0/3342129487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fe8f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:29.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 -- 172.21.15.38:0/3342129487 >> 172.21.15.38:0/3342129487 conn(0x7f68f40f63b0 msgr2=0x7f68f40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:29.426 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 -- 172.21.15.38:0/3342129487 shutdown_connections 2023-11-15T04:08:29.426 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.421+0000 7f68fab52700 1 -- 172.21.15.38:0/3342129487 wait complete. 2023-11-15T04:08:29.426 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68fab52700 1 Processor -- start 2023-11-15T04:08:29.426 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68fab52700 1 -- start start 2023-11-15T04:08:29.427 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68fab52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:29.427 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68fab52700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40fa7f0 con 0x7f68f40704a0 2023-11-15T04:08:29.427 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68f88ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:29.427 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68f88ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57928/0 (socket says 172.21.15.38:57928) 2023-11-15T04:08:29.427 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.422+0000 7f68f88ee700 1 -- 172.21.15.38:0/1139234866 learned_addr learned my addr 172.21.15.38:0/1139234866 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:29.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68f88ee700 1 -- 172.21.15.38:0/1139234866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e800b200 con 0x7f68f40704a0 2023-11-15T04:08:29.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68f88ee700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f68e800e040 tx=0x7f68e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:29.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e800d9f0 con 0x7f68f40704a0 2023-11-15T04:08:29.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f68e801a9f0 con 0x7f68f40704a0 2023-11-15T04:08:29.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e801b8c0 con 0x7f68f40704a0 2023-11-15T04:08:29.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f40fa990 con 0x7f68f40704a0 2023-11-15T04:08:29.430 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.423+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f40fada0 con 0x7f68f40704a0 2023-11-15T04:08:29.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.424+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f68e800f070 con 0x7f68f40704a0 2023-11-15T04:08:29.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.424+0000 7f68f17fa700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 0x7f68d8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:29.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.424+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f68e80412c0 con 0x7f68f40704a0 2023-11-15T04:08:29.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.424+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f4045ad0 con 0x7f68f40704a0 2023-11-15T04:08:29.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.424+0000 7f68f3fff700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 0x7f68d8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:29.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.425+0000 7f68f3fff700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 0x7f68d8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.429+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68e8016070 con 0x7f68f40704a0 2023-11-15T04:08:29.625 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.624+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f68f4104870 con 0x7f68d802fdb0 2023-11-15T04:08:29.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.629+0000 7f68f17fa700 1 -- 172.21.15.38:0/1139234866 <== mgr.14116 v2:172.21.15.38:6800/2226868937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f68f4104870 con 0x7f68d802fdb0 2023-11-15T04:08:29.629 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T04:08:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.630+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 msgr2=0x7f68d8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.630+0000 7f68fab52700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 0x7f68d8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).stop 2023-11-15T04:08:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.630+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 msgr2=0x7f68f40fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.630+0000 7f68fab52700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f68e800e040 tx=0x7f68e8006ee0).stop 2023-11-15T04:08:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 shutdown_connections 2023-11-15T04:08:29.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f68d802fdb0 0x7f68d8032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:29.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 --2- 172.21.15.38:0/1139234866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68f40704a0 0x7f68f40fc370 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:29.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 >> 172.21.15.38:0/1139234866 conn(0x7f68f40f63b0 msgr2=0x7f68f40f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:29.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 shutdown_connections 2023-11-15T04:08:29.633 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:29.631+0000 7f68fab52700 1 -- 172.21.15.38:0/1139234866 wait complete. 2023-11-15T04:08:29.666 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:27.697840+0000 mgr.smithi038.bnbogz (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:29.666 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:27.698668+0000 mgr.smithi038.bnbogz (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T04:08:29.667 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:28.630660+0000 mon.smithi038 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling mgr prometheus module... 2023-11-15T04:08:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T04:08:30.391 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.388+0000 7f4dbc6d8700 1 Processor -- start 2023-11-15T04:08:30.391 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.388+0000 7f4dbc6d8700 1 -- start start 2023-11-15T04:08:30.391 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.388+0000 7f4dbc6d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:30.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dbc6d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40fdc80 con 0x7f4db40fd370 2023-11-15T04:08:30.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dba474700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:30.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dba474700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57936/0 (socket says 172.21.15.38:57936) 2023-11-15T04:08:30.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dba474700 1 -- 172.21.15.38:0/1021260533 learned_addr learned my addr 172.21.15.38:0/1021260533 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:30.392 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dba474700 1 -- 172.21.15.38:0/1021260533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40fddc0 con 0x7f4db40fd370 2023-11-15T04:08:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4dba474700 1 --2- 172.21.15.38:0/1021260533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4da800b5f0 tx=0x7f4da8006ee0).ready entity=mon.0 client_cookie=efd14de2ed5f74bf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4db9472700 1 -- 172.21.15.38:0/1021260533 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da800d600 con 0x7f4db40fd370 2023-11-15T04:08:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4db9472700 1 -- 172.21.15.38:0/1021260533 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4da800dbd0 con 0x7f4db40fd370 2023-11-15T04:08:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.389+0000 7f4db9472700 1 -- 172.21.15.38:0/1021260533 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da8011c00 con 0x7f4db40fd370 2023-11-15T04:08:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/1021260533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 msgr2=0x7f4db40fd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:30.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/1021260533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4da800b5f0 tx=0x7f4da8006ee0).stop 2023-11-15T04:08:30.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/1021260533 shutdown_connections 2023-11-15T04:08:30.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/1021260533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db40fd790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:30.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/1021260533 >> 172.21.15.38:0/1021260533 conn(0x7f4db40f5b10 msgr2=0x7f4db40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:30.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/1021260533 shutdown_connections 2023-11-15T04:08:30.395 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.390+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/1021260533 wait complete. 2023-11-15T04:08:30.397 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.391+0000 7f4dbc6d8700 1 Processor -- start 2023-11-15T04:08:30.397 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.391+0000 7f4dbc6d8700 1 -- start start 2023-11-15T04:08:30.397 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.391+0000 7f4dbc6d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:30.397 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.391+0000 7f4dbc6d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db410aef0 con 0x7f4db40fd370 2023-11-15T04:08:30.398 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.391+0000 7f4dba474700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:30.398 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dba474700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57950/0 (socket says 172.21.15.38:57950) 2023-11-15T04:08:30.398 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dba474700 1 -- 172.21.15.38:0/2705114665 learned_addr learned my addr 172.21.15.38:0/2705114665 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:30.398 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dba474700 1 -- 172.21.15.38:0/2705114665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da800b200 con 0x7f4db40fd370 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dba474700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4da8002f40 tx=0x7f4da8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da801b410 con 0x7f4db40fd370 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4da801b9e0 con 0x7f4db40fd370 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db410b0f0 con 0x7f4db40fd370 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da801a8b0 con 0x7f4db40fd370 2023-11-15T04:08:30.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.392+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db410b500 con 0x7f4db40fd370 2023-11-15T04:08:30.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.393+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4da800f070 con 0x7f4db40fd370 2023-11-15T04:08:30.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.393+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db4045ad0 con 0x7f4db40fd370 2023-11-15T04:08:30.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.394+0000 7f4da6ffd700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 0x7f4d9c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:30.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.394+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4da80411f0 con 0x7f4db40fd370 2023-11-15T04:08:30.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.395+0000 7f4db9c73700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 0x7f4d9c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:30.401 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.395+0000 7f4db9c73700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 0x7f4d9c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4db00096a0 tx=0x7f4db0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:30.401 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.400+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4da8019950 con 0x7f4db40fd370 2023-11-15T04:08:30.608 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:30.607+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f4db40fa6e0 con 0x7f4db40fd370 2023-11-15T04:08:30.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:28.627553+0000 mgr.smithi038.bnbogz (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:30.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:28.628149+0000 mgr.smithi038.bnbogz (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T04:08:30.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:29.629606+0000 mon.smithi038 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.38:0/1848755973' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:31.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.398+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f4da801bc10 con 0x7f4db40fd370 2023-11-15T04:08:31.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.399+0000 7f4da6ffd700 1 -- 172.21.15.38:0/2705114665 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f4da8023970 con 0x7f4db40fd370 2023-11-15T04:08:31.403 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 msgr2=0x7f4d9c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:31.404 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 0x7f4d9c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4db00096a0 tx=0x7f4db0006b40).stop 2023-11-15T04:08:31.404 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 msgr2=0x7f4db410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:31.404 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4da8002f40 tx=0x7f4da8006ee0).stop 2023-11-15T04:08:31.405 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 shutdown_connections 2023-11-15T04:08:31.405 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7f4d9c02fd50 0x7f4d9c032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:31.405 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 --2- 172.21.15.38:0/2705114665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4db40fd370 0x7f4db410aa00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:31.406 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.403+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 >> 172.21.15.38:0/2705114665 conn(0x7f4db40f5b10 msgr2=0x7f4db40ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:31.406 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.404+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 shutdown_connections 2023-11-15T04:08:31.406 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:31.404+0000 7f4dbc6d8700 1 -- 172.21.15.38:0/2705114665 wait complete. 2023-11-15T04:08:31.717 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:29.625949+0000 mgr.smithi038.bnbogz (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:31.717 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:29.626641+0000 mgr.smithi038.bnbogz (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T04:08:31.717 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:30.608467+0000 mon.smithi038 (mon.0) 60 : audit [INF] from='client.? 172.21.15.38:0/2705114665' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T04:08:31.780 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying prometheus service with default placement... 2023-11-15T04:08:31.781 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T04:08:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.297+0000 7fd36cb8c700 1 Processor -- start 2023-11-15T04:08:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.297+0000 7fd36cb8c700 1 -- start start 2023-11-15T04:08:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36cb8c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36cb8c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680fc750 con 0x7fd3680703b0 2023-11-15T04:08:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57984/0 (socket says 172.21.15.38:57984) 2023-11-15T04:08:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36659c700 1 -- 172.21.15.38:0/1957586088 learned_addr learned my addr 172.21.15.38:0/1957586088 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.298+0000 7fd36659c700 1 -- 172.21.15.38:0/1957586088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368101250 con 0x7fd3680703b0 2023-11-15T04:08:32.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.299+0000 7fd36659c700 1 --2- 172.21.15.38:0/1957586088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd35800b5f0 tx=0x7fd358006ee0).ready entity=mon.0 client_cookie=d30876a2d52b1d13 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:32.306 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.299+0000 7fd36559a700 1 -- 172.21.15.38:0/1957586088 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd35800d600 con 0x7fd3680703b0 2023-11-15T04:08:32.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.299+0000 7fd36559a700 1 -- 172.21.15.38:0/1957586088 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd35800dbd0 con 0x7fd3680703b0 2023-11-15T04:08:32.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.299+0000 7fd36559a700 1 -- 172.21.15.38:0/1957586088 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd358011c00 con 0x7fd3680703b0 2023-11-15T04:08:32.307 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 -- 172.21.15.38:0/1957586088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 msgr2=0x7fd368100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:32.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/1957586088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd35800b5f0 tx=0x7fd358006ee0).stop 2023-11-15T04:08:32.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 -- 172.21.15.38:0/1957586088 shutdown_connections 2023-11-15T04:08:32.308 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/1957586088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd368100d60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:32.309 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 -- 172.21.15.38:0/1957586088 >> 172.21.15.38:0/1957586088 conn(0x7fd3680f63c0 msgr2=0x7fd3680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:32.309 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 -- 172.21.15.38:0/1957586088 shutdown_connections 2023-11-15T04:08:32.309 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 -- 172.21.15.38:0/1957586088 wait complete. 2023-11-15T04:08:32.309 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.300+0000 7fd36cb8c700 1 Processor -- start 2023-11-15T04:08:32.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36cb8c700 1 -- start start 2023-11-15T04:08:32.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36cb8c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:32.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36cb8c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36810d100 con 0x7fd3680703b0 2023-11-15T04:08:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58000/0 (socket says 172.21.15.38:58000) 2023-11-15T04:08:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36659c700 1 -- 172.21.15.38:0/936509308 learned_addr learned my addr 172.21.15.38:0/936509308 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36659c700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35800b200 con 0x7fd3680703b0 2023-11-15T04:08:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd36659c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd35800b190 tx=0x7fd358006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.301+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd35801b410 con 0x7fd3680703b0 2023-11-15T04:08:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.302+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd35801b9e0 con 0x7fd3680703b0 2023-11-15T04:08:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.302+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36810d300 con 0x7fd3680703b0 2023-11-15T04:08:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.302+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd35801a880 con 0x7fd3680703b0 2023-11-15T04:08:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.302+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36810d710 con 0x7fd3680703b0 2023-11-15T04:08:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.302+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd35800f070 con 0x7fd3680703b0 2023-11-15T04:08:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.303+0000 7fd356ffd700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.303+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd3580412a0 con 0x7fd3680703b0 2023-11-15T04:08:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.303+0000 7fd365d9b700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2226868937 2023-11-15T04:08:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.303+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:08:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.303+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368045ad0 con 0x7fd3680703b0 2023-11-15T04:08:32.315 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.307+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3580224f0 con 0x7fd3680703b0 2023-11-15T04:08:32.505 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.504+0000 7fd365d9b700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2226868937 2023-11-15T04:08:32.505 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.504+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:08:32.506 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.505+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd368106b10 con 0x7fd34c02fe00 2023-11-15T04:08:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:31.400550+0000 mon.smithi038 (mon.0) 61 : audit [INF] from='client.? 172.21.15.38:0/2705114665' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T04:08:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:31.400644+0000 mon.smithi038 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi038.bnbogz(active, since 17s) 2023-11-15T04:08:32.905 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.905+0000 7fd365d9b700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2226868937 2023-11-15T04:08:32.906 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:32.905+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:08:33.706 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:33.706+0000 7fd365d9b700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2226868937 2023-11-15T04:08:33.707 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:33.706+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:08:35.310 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:35.307+0000 7fd365d9b700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2226868937 2023-11-15T04:08:35.311 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:35.307+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:08:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:37.392+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd358025460 con 0x7fd3680703b0 2023-11-15T04:08:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:37.392+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 msgr2=0x7fd34c0322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:08:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:37.392+0000 7fd356ffd700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:37.650 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:37.035998+0000 mon.smithi038 (mon.0) 63 : cluster [INF] Active manager daemon smithi038.bnbogz restarted 2023-11-15T04:08:37.651 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:37.036564+0000 mon.smithi038 (mon.0) 64 : cluster [INF] Activating manager daemon smithi038.bnbogz 2023-11-15T04:08:37.651 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:37.042824+0000 mon.smithi038 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T04:08:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.396+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd358043000 con 0x7fd3680703b0 2023-11-15T04:08:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.396+0000 7fd356ffd700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 0x7fd34c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.397+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 --> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd368106b10 con 0x7fd34c033c20 2023-11-15T04:08:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.397+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 0x7fd34c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:38.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.398+0000 7fd365d9b700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 0x7fd34c036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd35c006fd0 tx=0x7fd35c006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.408+0000 7fd356ffd700 1 -- 172.21.15.38:0/936509308 <== mgr.14144 v2:172.21.15.38:6800/27986150 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd368106b10 con 0x7fd34c033c20 2023-11-15T04:08:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T04:08:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 msgr2=0x7fd34c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:38.412 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 0x7fd34c036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd35c006fd0 tx=0x7fd35c006d30).stop 2023-11-15T04:08:38.412 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 msgr2=0x7fd36810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:38.412 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd35800b190 tx=0x7fd358006ee0).stop 2023-11-15T04:08:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 shutdown_connections 2023-11-15T04:08:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fd34c033c20 0x7fd34c036010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3680703b0 0x7fd36810cc10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 --2- 172.21.15.38:0/936509308 >> [v2:172.21.15.38:6800/2226868937,v1:172.21.15.38:6801/2226868937] conn(0x7fd34c02fe00 0x7fd34c0322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 >> 172.21.15.38:0/936509308 conn(0x7fd3680f63c0 msgr2=0x7fd3680f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 shutdown_connections 2023-11-15T04:08:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:38.411+0000 7fd36cb8c700 1 -- 172.21.15.38:0/936509308 wait complete. 2023-11-15T04:08:38.750 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:37.394748+0000 mon.smithi038 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi038.bnbogz(active, starting, since 0.358275s) 2023-11-15T04:08:38.750 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.396116+0000 mon.smithi038 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2023-11-15T04:08:38.750 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.396890+0000 mon.smithi038 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mgr metadata", "who": "smithi038.bnbogz", "id": "smithi038.bnbogz"}]: dispatch 2023-11-15T04:08:38.751 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.397901+0000 mon.smithi038 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:08:38.751 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.398187+0000 mon.smithi038 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:08:38.751 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.398393+0000 mon.smithi038 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:08:38.751 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:37.413033+0000 mon.smithi038 (mon.0) 72 : cluster [INF] Manager daemon smithi038.bnbogz is now available 2023-11-15T04:08:38.751 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.531671+0000 mon.smithi038 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:38.752 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.533283+0000 mon.smithi038 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:38.752 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.550826+0000 mon.smithi038 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:08:38.752 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:37.560177+0000 mon.smithi038 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:08:38.753 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying grafana service with default placement... 2023-11-15T04:08:38.753 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T04:08:39.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbaba4d4700 1 Processor -- start 2023-11-15T04:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbaba4d4700 1 -- start start 2023-11-15T04:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbaba4d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbaba4d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40fdbb0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44312/0 (socket says 172.21.15.38:44312) 2023-11-15T04:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbab3fff700 1 -- 172.21.15.38:0/2261995194 learned_addr learned my addr 172.21.15.38:0/2261995194 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbab3fff700 1 -- 172.21.15.38:0/2261995194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40fdcf0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.197+0000 7fbab3fff700 1 --2- 172.21.15.38:0/2261995194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbaa400b5f0 tx=0x7fbaa4006ee0).ready entity=mon.0 client_cookie=9fba6afd90a556cf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.198+0000 7fbab2ffd700 1 -- 172.21.15.38:0/2261995194 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa400d600 con 0x7fbab40fd2a0 2023-11-15T04:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.198+0000 7fbab2ffd700 1 -- 172.21.15.38:0/2261995194 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbaa400dbd0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.198+0000 7fbab2ffd700 1 -- 172.21.15.38:0/2261995194 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa4011c00 con 0x7fbab40fd2a0 2023-11-15T04:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.198+0000 7fbaba4d4700 1 -- 172.21.15.38:0/2261995194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 msgr2=0x7fbab40fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.198+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/2261995194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbaa400b5f0 tx=0x7fbaa4006ee0).stop 2023-11-15T04:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 -- 172.21.15.38:0/2261995194 shutdown_connections 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/2261995194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab40fd6c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 -- 172.21.15.38:0/2261995194 >> 172.21.15.38:0/2261995194 conn(0x7fbab40f5b40 msgr2=0x7fbab40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 -- 172.21.15.38:0/2261995194 shutdown_connections 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 -- 172.21.15.38:0/2261995194 wait complete. 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 Processor -- start 2023-11-15T04:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.199+0000 7fbaba4d4700 1 -- start start 2023-11-15T04:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbaba4d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbaba4d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab410af50 con 0x7fbab40fd2a0 2023-11-15T04:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44314/0 (socket says 172.21.15.38:44314) 2023-11-15T04:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab3fff700 1 -- 172.21.15.38:0/3521031890 learned_addr learned my addr 172.21.15.38:0/3521031890 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:39.207 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab3fff700 1 -- 172.21.15.38:0/3521031890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa400b200 con 0x7fbab40fd2a0 2023-11-15T04:08:39.207 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab3fff700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fbaa4006c70 tx=0x7fbaa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:39.207 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa400d9f0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.207 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbaa401a9f0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.207 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab410b150 con 0x7fbab40fd2a0 2023-11-15T04:08:39.208 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa401b890 con 0x7fbab40fd2a0 2023-11-15T04:08:39.208 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.200+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab410b560 con 0x7fbab40fd2a0 2023-11-15T04:08:39.208 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.201+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fbaa400f070 con 0x7fbab40fd2a0 2023-11-15T04:08:39.208 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.201+0000 7fbab0ff9700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 0x7fbaa00321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:39.208 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.201+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbaa4042120 con 0x7fbab40fd2a0 2023-11-15T04:08:39.209 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.202+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4045ad0 con 0x7fbab40fd2a0 2023-11-15T04:08:39.209 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.202+0000 7fbab37fe700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 0x7fbaa00321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:39.209 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.202+0000 7fbab37fe700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 0x7fbaa00321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbaa80096a0 tx=0x7fbaa8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:39.210 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.206+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbaa4016070 con 0x7fbab40fd2a0 2023-11-15T04:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.400+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 --> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fbab4104570 con 0x7fbaa002fd10 2023-11-15T04:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.406+0000 7fbab0ff9700 1 -- 172.21.15.38:0/3521031890 <== mgr.14144 v2:172.21.15.38:6800/27986150 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fbab4104570 con 0x7fbaa002fd10 2023-11-15T04:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T04:08:39.410 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 msgr2=0x7fbaa00321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:39.410 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 0x7fbaa00321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbaa80096a0 tx=0x7fbaa8006b40).stop 2023-11-15T04:08:39.410 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 msgr2=0x7fbab410aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:39.410 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fbaa4006c70 tx=0x7fbaa4006ee0).stop 2023-11-15T04:08:39.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 shutdown_connections 2023-11-15T04:08:39.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fbaa002fd10 0x7fbaa00321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:39.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 --2- 172.21.15.38:0/3521031890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbab40fd2a0 0x7fbab410aa60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:39.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 >> 172.21.15.38:0/3521031890 conn(0x7fbab40f5b40 msgr2=0x7fbab40ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:39.411 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.408+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 shutdown_connections 2023-11-15T04:08:39.412 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:39.409+0000 7fbaba4d4700 1 -- 172.21.15.38:0/3521031890 wait complete. 2023-11-15T04:08:39.749 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:38.400017+0000 mon.smithi038 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi038.bnbogz(active, since 1.36354s) 2023-11-15T04:08:39.750 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:38.401092+0000 mgr.smithi038.bnbogz (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T04:08:39.750 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:38.409336+0000 mon.smithi038 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying node-exporter service with default placement... 2023-11-15T04:08:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T04:08:40.222 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35bc446700 1 Processor -- start 2023-11-15T04:08:40.222 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35bc446700 1 -- start start 2023-11-15T04:08:40.222 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35bc446700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:40.223 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35bc446700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40fdc90 con 0x7f35b40fd380 2023-11-15T04:08:40.223 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35ba1e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:40.223 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35ba1e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44316/0 (socket says 172.21.15.38:44316) 2023-11-15T04:08:40.226 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.221+0000 7f35ba1e2700 1 -- 172.21.15.38:0/1995698569 learned_addr learned my addr 172.21.15.38:0/1995698569 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:40.226 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.222+0000 7f35ba1e2700 1 -- 172.21.15.38:0/1995698569 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40fddd0 con 0x7f35b40fd380 2023-11-15T04:08:40.227 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.222+0000 7f35ba1e2700 1 --2- 172.21.15.38:0/1995698569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f35b000b5f0 tx=0x7f35b0006ee0).ready entity=mon.0 client_cookie=2609b298753253e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:40.227 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.222+0000 7f35b91e0700 1 -- 172.21.15.38:0/1995698569 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b000d600 con 0x7f35b40fd380 2023-11-15T04:08:40.227 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.222+0000 7f35b91e0700 1 -- 172.21.15.38:0/1995698569 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f35b000dbd0 con 0x7f35b40fd380 2023-11-15T04:08:40.227 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.222+0000 7f35b91e0700 1 -- 172.21.15.38:0/1995698569 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b0011c00 con 0x7f35b40fd380 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- 172.21.15.38:0/1995698569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 msgr2=0x7f35b40fd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 --2- 172.21.15.38:0/1995698569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f35b000b5f0 tx=0x7f35b0006ee0).stop 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- 172.21.15.38:0/1995698569 shutdown_connections 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 --2- 172.21.15.38:0/1995698569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fd7a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- 172.21.15.38:0/1995698569 >> 172.21.15.38:0/1995698569 conn(0x7f35b40f5b40 msgr2=0x7f35b40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:40.228 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- 172.21.15.38:0/1995698569 shutdown_connections 2023-11-15T04:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- 172.21.15.38:0/1995698569 wait complete. 2023-11-15T04:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 Processor -- start 2023-11-15T04:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.223+0000 7f35bc446700 1 -- start start 2023-11-15T04:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35bc446700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35bc446700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40fb0d0 con 0x7f35b40fd380 2023-11-15T04:08:40.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35ba1e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:40.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35ba1e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44320/0 (socket says 172.21.15.38:44320) 2023-11-15T04:08:40.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35ba1e2700 1 -- 172.21.15.38:0/1502765720 learned_addr learned my addr 172.21.15.38:0/1502765720 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:40.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35ba1e2700 1 -- 172.21.15.38:0/1502765720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b000b200 con 0x7f35b40fd380 2023-11-15T04:08:40.230 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35ba1e2700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f35b0002f40 tx=0x7f35b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b001b410 con 0x7f35b40fd380 2023-11-15T04:08:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f35b001b9e0 con 0x7f35b40fd380 2023-11-15T04:08:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b40fb270 con 0x7f35b40fd380 2023-11-15T04:08:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b001a8b0 con 0x7f35b40fd380 2023-11-15T04:08:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.224+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b40fb680 con 0x7f35b40fd380 2023-11-15T04:08:40.232 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f35b000f070 con 0x7f35b40fd380 2023-11-15T04:08:40.232 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b4045ad0 con 0x7f35b40fd380 2023-11-15T04:08:40.232 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35a2ffd700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 0x7f359c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:40.232 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f35b0041450 con 0x7f35b40fd380 2023-11-15T04:08:40.232 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35b99e1700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 0x7f359c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:40.233 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.226+0000 7f35b99e1700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 0x7f359c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f35a40096a0 tx=0x7f35a4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:40.233 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.230+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35b0019910 con 0x7f35b40fd380 2023-11-15T04:08:40.425 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.424+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 --> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f35b40fa810 con 0x7f359c02fe60 2023-11-15T04:08:40.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.428+0000 7f35a2ffd700 1 -- 172.21.15.38:0/1502765720 <== mgr.14144 v2:172.21.15.38:6800/27986150 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f35b40fa810 con 0x7f359c02fe60 2023-11-15T04:08:40.429 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T04:08:40.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 msgr2=0x7f359c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:40.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 0x7f359c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f35a40096a0 tx=0x7f35a4006b40).stop 2023-11-15T04:08:40.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 msgr2=0x7f35b40fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:40.431 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f35b0002f40 tx=0x7f35b0006ee0).stop 2023-11-15T04:08:40.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 shutdown_connections 2023-11-15T04:08:40.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f359c02fe60 0x7f359c032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:40.432 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 --2- 172.21.15.38:0/1502765720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35b40fd380 0x7f35b40fcc50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:40.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 >> 172.21.15.38:0/1502765720 conn(0x7f35b40f5b40 msgr2=0x7f35b40ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:40.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 shutdown_connections 2023-11-15T04:08:40.433 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:40.430+0000 7f35bc446700 1 -- 172.21.15.38:0/1502765720 wait complete. 2023-11-15T04:08:40.732 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:39.401603+0000 mgr.smithi038.bnbogz (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:40.733 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:39.402152+0000 mgr.smithi038.bnbogz (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T04:08:40.733 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:39.407391+0000 mon.smithi038 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:40.733 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:39.417025+0000 mon.smithi038 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi038.bnbogz(active, since 2s) 2023-11-15T04:08:40.788 INFO:teuthology.orchestra.run.smithi038.stderr:Deploying alertmanager service with default placement... 2023-11-15T04:08:40.788 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T04:08:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.153+0000 7fe4ae87e700 1 Processor -- start 2023-11-15T04:08:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.153+0000 7fe4ae87e700 1 -- start start 2023-11-15T04:08:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4ae87e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:41.160 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4ae87e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80fb490 con 0x7fe4a80fab80 2023-11-15T04:08:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44332/0 (socket says 172.21.15.38:44332) 2023-11-15T04:08:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a7fff700 1 -- 172.21.15.38:0/3057669485 learned_addr learned my addr 172.21.15.38:0/3057669485 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a7fff700 1 -- 172.21.15.38:0/3057669485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80fb5d0 con 0x7fe4a80fab80 2023-11-15T04:08:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a7fff700 1 --2- 172.21.15.38:0/3057669485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe49800b870 tx=0x7fe498006f70).ready entity=mon.0 client_cookie=a894c78ddd21d183 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a6ffd700 1 -- 172.21.15.38:0/3057669485 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe49800da20 con 0x7fe4a80fab80 2023-11-15T04:08:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.154+0000 7fe4a6ffd700 1 -- 172.21.15.38:0/3057669485 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe498010470 con 0x7fe4a80fab80 2023-11-15T04:08:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4a6ffd700 1 -- 172.21.15.38:0/3057669485 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe49800f5b0 con 0x7fe4a80fab80 2023-11-15T04:08:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 -- 172.21.15.38:0/3057669485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 msgr2=0x7fe4a80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/3057669485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe49800b870 tx=0x7fe498006f70).stop 2023-11-15T04:08:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 -- 172.21.15.38:0/3057669485 shutdown_connections 2023-11-15T04:08:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/3057669485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a80fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 -- 172.21.15.38:0/3057669485 >> 172.21.15.38:0/3057669485 conn(0x7fe4a80f63c0 msgr2=0x7fe4a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 -- 172.21.15.38:0/3057669485 shutdown_connections 2023-11-15T04:08:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.155+0000 7fe4ae87e700 1 -- 172.21.15.38:0/3057669485 wait complete. 2023-11-15T04:08:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4ae87e700 1 Processor -- start 2023-11-15T04:08:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4ae87e700 1 -- start start 2023-11-15T04:08:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4ae87e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:41.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4ae87e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a8108c60 con 0x7fe4a80fab80 2023-11-15T04:08:41.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:41.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44340/0 (socket says 172.21.15.38:44340) 2023-11-15T04:08:41.165 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.156+0000 7fe4a7fff700 1 -- 172.21.15.38:0/625855325 learned_addr learned my addr 172.21.15.38:0/625855325 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:41.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4a7fff700 1 -- 172.21.15.38:0/625855325 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49800b480 con 0x7fe4a80fab80 2023-11-15T04:08:41.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4a7fff700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe498006800 tx=0x7fe498006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:41.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe49800f820 con 0x7fe4a80fab80 2023-11-15T04:08:41.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe49800fdf0 con 0x7fe4a80fab80 2023-11-15T04:08:41.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe498019d50 con 0x7fe4a80fab80 2023-11-15T04:08:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a8108e60 con 0x7fe4a80fab80 2023-11-15T04:08:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.157+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a8109270 con 0x7fe4a80fab80 2023-11-15T04:08:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.158+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe498016030 con 0x7fe4a80fab80 2023-11-15T04:08:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.158+0000 7fe4a4ff9700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 0x7fe494032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.158+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe4980442b0 con 0x7fe4a80fab80 2023-11-15T04:08:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.158+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a8045ad0 con 0x7fe4a80fab80 2023-11-15T04:08:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.158+0000 7fe4a77fe700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 0x7fe494032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.159+0000 7fe4a77fe700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 0x7fe494032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe49c0096a0 tx=0x7fe49c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.163+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe498014080 con 0x7fe4a80fab80 2023-11-15T04:08:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.356+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 --> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4a8109b50 con 0x7fe49402feb0 2023-11-15T04:08:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.362+0000 7fe4a4ff9700 1 -- 172.21.15.38:0/625855325 <== mgr.14144 v2:172.21.15.38:6800/27986150 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fe4a8109b50 con 0x7fe49402feb0 2023-11-15T04:08:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T04:08:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.364+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 msgr2=0x7fe494032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.364+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 0x7fe494032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe49c0096a0 tx=0x7fe49c006b40).stop 2023-11-15T04:08:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.364+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 msgr2=0x7fe4a8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.364+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe498006800 tx=0x7fe498006f70).stop 2023-11-15T04:08:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 shutdown_connections 2023-11-15T04:08:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7fe49402feb0 0x7fe494032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 --2- 172.21.15.38:0/625855325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a80fab80 0x7fe4a8108770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 >> 172.21.15.38:0/625855325 conn(0x7fe4a80f63c0 msgr2=0x7fe4a80ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 shutdown_connections 2023-11-15T04:08:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:41.365+0000 7fe4ae87e700 1 -- 172.21.15.38:0/625855325 wait complete. 2023-11-15T04:08:41.682 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:40.426341+0000 mgr.smithi038.bnbogz (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:41.682 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:40.427047+0000 mgr.smithi038.bnbogz (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T04:08:41.683 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:40.429076+0000 mon.smithi038 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:41.683 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:41.363535+0000 mon.smithi038 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.38:0/530228469' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:41.713 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T04:08:42.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f239d5fd700 1 Processor -- start 2023-11-15T04:08:42.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f239d5fd700 1 -- start start 2023-11-15T04:08:42.144 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f239d5fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:42.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f239d5fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980fc5c0 con 0x7f23980fbcb0 2023-11-15T04:08:42.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f2396ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:42.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f2396ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44344/0 (socket says 172.21.15.38:44344) 2023-11-15T04:08:42.145 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f2396ffd700 1 -- 172.21.15.38:0/2554874639 learned_addr learned my addr 172.21.15.38:0/2554874639 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:42.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.143+0000 7f2396ffd700 1 -- 172.21.15.38:0/2554874639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980fc700 con 0x7f23980fbcb0 2023-11-15T04:08:42.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f2396ffd700 1 --2- 172.21.15.38:0/2554874639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f238800b5f0 tx=0x7f2388006ee0).ready entity=mon.0 client_cookie=9b27696204554f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:42.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f23967fc700 1 -- 172.21.15.38:0/2554874639 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f238800d600 con 0x7f23980fbcb0 2023-11-15T04:08:42.149 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f23967fc700 1 -- 172.21.15.38:0/2554874639 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f238800dbd0 con 0x7f23980fbcb0 2023-11-15T04:08:42.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f23967fc700 1 -- 172.21.15.38:0/2554874639 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2388011c00 con 0x7f23980fbcb0 2023-11-15T04:08:42.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 -- 172.21.15.38:0/2554874639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 msgr2=0x7f23980fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:42.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2554874639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f238800b5f0 tx=0x7f2388006ee0).stop 2023-11-15T04:08:42.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 -- 172.21.15.38:0/2554874639 shutdown_connections 2023-11-15T04:08:42.150 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2554874639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f23980fc0d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:42.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 -- 172.21.15.38:0/2554874639 >> 172.21.15.38:0/2554874639 conn(0x7f23980f74b0 msgr2=0x7f23980f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:42.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.144+0000 7f239d5fd700 1 -- 172.21.15.38:0/2554874639 shutdown_connections 2023-11-15T04:08:42.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.145+0000 7f239d5fd700 1 -- 172.21.15.38:0/2554874639 wait complete. 2023-11-15T04:08:42.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.145+0000 7f239d5fd700 1 Processor -- start 2023-11-15T04:08:42.151 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.145+0000 7f239d5fd700 1 -- start start 2023-11-15T04:08:42.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f239d5fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:42.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f239d5fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239810af10 con 0x7f23980fbcb0 2023-11-15T04:08:42.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f2396ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:42.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f2396ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44356/0 (socket says 172.21.15.38:44356) 2023-11-15T04:08:42.152 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f2396ffd700 1 -- 172.21.15.38:0/2926879705 learned_addr learned my addr 172.21.15.38:0/2926879705 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f2396ffd700 1 -- 172.21.15.38:0/2926879705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238800b200 con 0x7f23980fbcb0 2023-11-15T04:08:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f2396ffd700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f238800b1d0 tx=0x7f2388006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f238801b410 con 0x7f23980fbcb0 2023-11-15T04:08:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f238801b9e0 con 0x7f23980fbcb0 2023-11-15T04:08:42.153 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f238801a840 con 0x7f23980fbcb0 2023-11-15T04:08:42.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239810b110 con 0x7f23980fbcb0 2023-11-15T04:08:42.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.146+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239810b520 con 0x7f23980fbcb0 2023-11-15T04:08:42.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.147+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f238800f070 con 0x7f23980fbcb0 2023-11-15T04:08:42.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.147+0000 7f238ffff700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 0x7f2378032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:42.154 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.147+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2388041390 con 0x7f23980fbcb0 2023-11-15T04:08:42.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.147+0000 7f238e7fc700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 0x7f2378032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:42.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.148+0000 7f238e7fc700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 0x7f2378032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:42.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.148+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2398059100 con 0x7f23980fbcb0 2023-11-15T04:08:42.155 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.152+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2388022d70 con 0x7f23980fbcb0 2023-11-15T04:08:42.319 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.319+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f2398104890 con 0x7f23980fbcb0 2023-11-15T04:08:42.325 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.324+0000 7f238ffff700 1 -- 172.21.15.38:0/2926879705 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2388022310 con 0x7f23980fbcb0 2023-11-15T04:08:42.327 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 msgr2=0x7f2378032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 0x7f2378032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).stop 2023-11-15T04:08:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 msgr2=0x7f239810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f238800b1d0 tx=0x7f2388006ee0).stop 2023-11-15T04:08:42.329 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 shutdown_connections 2023-11-15T04:08:42.329 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f237802feb0 0x7f2378032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:42.329 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 --2- 172.21.15.38:0/2926879705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23980fbcb0 0x7f239810aa20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:42.330 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 >> 172.21.15.38:0/2926879705 conn(0x7f23980f74b0 msgr2=0x7f23980f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:42.330 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 shutdown_connections 2023-11-15T04:08:42.330 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:42.327+0000 7f239d5fd700 1 -- 172.21.15.38:0/2926879705 wait complete. 2023-11-15T04:08:42.632 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:41.358147+0000 mgr.smithi038.bnbogz (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:42.632 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:08:41.358699+0000 mgr.smithi038.bnbogz (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T04:08:42.633 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:42.325689+0000 mon.smithi038 (mon.0) 83 : audit [INF] from='client.? 172.21.15.38:0/2926879705' entity='client.admin' 2023-11-15T04:08:42.712 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T04:08:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa38dd700 1 Processor -- start 2023-11-15T04:08:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa38dd700 1 -- start start 2023-11-15T04:08:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa38dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa38dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c0fb490 con 0x7f6a9c0fab80 2023-11-15T04:08:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa1679700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:43.116 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.112+0000 7f6aa1679700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44366/0 (socket says 172.21.15.38:44366) 2023-11-15T04:08:43.116 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6aa1679700 1 -- 172.21.15.38:0/1594632511 learned_addr learned my addr 172.21.15.38:0/1594632511 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:43.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6aa1679700 1 -- 172.21.15.38:0/1594632511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c0fb5d0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6aa1679700 1 --2- 172.21.15.38:0/1594632511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f6a9800b5f0 tx=0x7f6a98006ee0).ready entity=mon.0 client_cookie=38f683c79cb22d8b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:43.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6a8bfff700 1 -- 172.21.15.38:0/1594632511 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a9800d600 con 0x7f6a9c0fab80 2023-11-15T04:08:43.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6a8bfff700 1 -- 172.21.15.38:0/1594632511 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6a9800dbd0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.117 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.113+0000 7f6a8bfff700 1 -- 172.21.15.38:0/1594632511 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a98011c00 con 0x7f6a9c0fab80 2023-11-15T04:08:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 -- 172.21.15.38:0/1594632511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 msgr2=0x7f6a9c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/1594632511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f6a9800b5f0 tx=0x7f6a98006ee0).stop 2023-11-15T04:08:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 -- 172.21.15.38:0/1594632511 shutdown_connections 2023-11-15T04:08:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/1594632511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c0fafa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 -- 172.21.15.38:0/1594632511 >> 172.21.15.38:0/1594632511 conn(0x7f6a9c0f63c0 msgr2=0x7f6a9c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:43.119 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 -- 172.21.15.38:0/1594632511 shutdown_connections 2023-11-15T04:08:43.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.114+0000 7f6aa38dd700 1 -- 172.21.15.38:0/1594632511 wait complete. 2023-11-15T04:08:43.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.115+0000 7f6aa38dd700 1 Processor -- start 2023-11-15T04:08:43.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.115+0000 7f6aa38dd700 1 -- start start 2023-11-15T04:08:43.120 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.115+0000 7f6aa38dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:43.121 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.115+0000 7f6aa38dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c10b4f0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.121 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa1679700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:43.121 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa1679700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44370/0 (socket says 172.21.15.38:44370) 2023-11-15T04:08:43.121 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa1679700 1 -- 172.21.15.38:0/3652390039 learned_addr learned my addr 172.21.15.38:0/3652390039 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:43.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa1679700 1 -- 172.21.15.38:0/3652390039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9800b200 con 0x7f6a9c0fab80 2023-11-15T04:08:43.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa1679700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6a9800b1d0 tx=0x7f6a98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:43.123 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a9801b410 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6a9801b9e0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c070450 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.116+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a9801a840 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.117+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c10fff0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.117+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6a9800f070 con 0x7f6a9c0fab80 2023-11-15T04:08:43.124 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.117+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9c045ad0 con 0x7f6a9c0fab80 2023-11-15T04:08:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.118+0000 7f6a89ffb700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 0x7f6a84032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.118+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6a98041350 con 0x7f6a9c0fab80 2023-11-15T04:08:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.118+0000 7f6aa0e78700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 0x7f6a84032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.118+0000 7f6aa0e78700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 0x7f6a84032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.122+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a98019950 con 0x7f6a9c0fab80 2023-11-15T04:08:43.289 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.288+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f6a9c0fc890 con 0x7f6a9c0fab80 2023-11-15T04:08:43.294 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.293+0000 7f6a89ffb700 1 -- 172.21.15.38:0/3652390039 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f6a98016030 con 0x7f6a9c0fab80 2023-11-15T04:08:43.296 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 msgr2=0x7f6a84032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:43.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 0x7f6a84032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).stop 2023-11-15T04:08:43.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 msgr2=0x7f6a9c10fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:43.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6a9800b1d0 tx=0x7f6a98006ee0).stop 2023-11-15T04:08:43.298 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 shutdown_connections 2023-11-15T04:08:43.298 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6a8402feb0 0x7f6a84032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:43.298 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 --2- 172.21.15.38:0/3652390039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a9c0fab80 0x7f6a9c10fb00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:43.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 >> 172.21.15.38:0/3652390039 conn(0x7f6a9c0f63c0 msgr2=0x7f6a9c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:43.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 shutdown_connections 2023-11-15T04:08:43.299 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:43.296+0000 7f6aa38dd700 1 -- 172.21.15.38:0/3652390039 wait complete. 2023-11-15T04:08:43.646 INFO:teuthology.orchestra.run.smithi038.stderr:Enabling the dashboard module... 2023-11-15T04:08:43.646 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T04:08:44.034 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.032+0000 7f6ddadc9700 1 Processor -- start 2023-11-15T04:08:44.035 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.032+0000 7f6ddadc9700 1 -- start start 2023-11-15T04:08:44.035 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6ddadc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:44.035 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6ddadc9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd40fc710 con 0x7f6dd40703f0 2023-11-15T04:08:44.035 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dd8b65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:44.035 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dd8b65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44378/0 (socket says 172.21.15.38:44378) 2023-11-15T04:08:44.036 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dd8b65700 1 -- 172.21.15.38:0/3833600431 learned_addr learned my addr 172.21.15.38:0/3833600431 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:44.036 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dd8b65700 1 -- 172.21.15.38:0/3833600431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd4101210 con 0x7f6dd40703f0 2023-11-15T04:08:44.036 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dd8b65700 1 --2- 172.21.15.38:0/3833600431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f6dd000b5f0 tx=0x7f6dd0006ee0).ready entity=mon.0 client_cookie=98892b8dc8f2a90e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:44.036 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.033+0000 7f6dcb7fe700 1 -- 172.21.15.38:0/3833600431 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd000d600 con 0x7f6dd40703f0 2023-11-15T04:08:44.039 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6dcb7fe700 1 -- 172.21.15.38:0/3833600431 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6dd000dbd0 con 0x7f6dd40703f0 2023-11-15T04:08:44.039 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6dcb7fe700 1 -- 172.21.15.38:0/3833600431 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd0011c00 con 0x7f6dd40703f0 2023-11-15T04:08:44.040 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3833600431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 msgr2=0x7f6dd4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:44.040 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3833600431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f6dd000b5f0 tx=0x7f6dd0006ee0).stop 2023-11-15T04:08:44.040 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3833600431 shutdown_connections 2023-11-15T04:08:44.040 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3833600431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd4100d20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:44.040 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3833600431 >> 172.21.15.38:0/3833600431 conn(0x7f6dd40f63c0 msgr2=0x7f6dd40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:44.041 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3833600431 shutdown_connections 2023-11-15T04:08:44.041 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.034+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3833600431 wait complete. 2023-11-15T04:08:44.042 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6ddadc9700 1 Processor -- start 2023-11-15T04:08:44.042 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6ddadc9700 1 -- start start 2023-11-15T04:08:44.042 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6ddadc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:44.042 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6ddadc9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd410d0b0 con 0x7f6dd40703f0 2023-11-15T04:08:44.043 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6dd8b65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:44.043 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6dd8b65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44386/0 (socket says 172.21.15.38:44386) 2023-11-15T04:08:44.043 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.035+0000 7f6dd8b65700 1 -- 172.21.15.38:0/3791402528 learned_addr learned my addr 172.21.15.38:0/3791402528 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:44.043 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6dd8b65700 1 -- 172.21.15.38:0/3791402528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd000b200 con 0x7f6dd40703f0 2023-11-15T04:08:44.044 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6dd8b65700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f6dd000b190 tx=0x7f6dd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:44.044 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd001b410 con 0x7f6dd40703f0 2023-11-15T04:08:44.044 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd410d2b0 con 0x7f6dd40703f0 2023-11-15T04:08:44.045 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6dd001b9e0 con 0x7f6dd40703f0 2023-11-15T04:08:44.045 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd001a880 con 0x7f6dd40703f0 2023-11-15T04:08:44.045 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.036+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd410d6c0 con 0x7f6dd40703f0 2023-11-15T04:08:44.045 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.037+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6dd000f070 con 0x7f6dd40703f0 2023-11-15T04:08:44.045 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.037+0000 7f6dc97fa700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 0x7f6dc0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:44.046 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.037+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6dd00422c0 con 0x7f6dd40703f0 2023-11-15T04:08:44.046 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.037+0000 7f6dcbfff700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 0x7f6dc0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:44.046 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.037+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd4059100 con 0x7f6dd40703f0 2023-11-15T04:08:44.046 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.038+0000 7f6dcbfff700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 0x7f6dc0032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6dc40096a0 tx=0x7f6dc4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:44.047 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.042+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dd0016070 con 0x7f6dd40703f0 2023-11-15T04:08:44.254 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.252+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f6dd4101dd0 con 0x7f6dd40703f0 2023-11-15T04:08:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.296+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6dd0025980 con 0x7f6dd40703f0 2023-11-15T04:08:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.297+0000 7f6dc97fa700 1 -- 172.21.15.38:0/3791402528 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f6dd0023540 con 0x7f6dd40703f0 2023-11-15T04:08:44.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 msgr2=0x7f6dc0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:44.301 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 0x7f6dc0032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6dc40096a0 tx=0x7f6dc4006b40).stop 2023-11-15T04:08:44.302 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 msgr2=0x7f6dd410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:44.302 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f6dd000b190 tx=0x7f6dd0006ee0).stop 2023-11-15T04:08:44.303 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 shutdown_connections 2023-11-15T04:08:44.303 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f6dc002fe60 0x7f6dc0032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:44.303 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 --2- 172.21.15.38:0/3791402528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dd40703f0 0x7f6dd410cbc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:44.303 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 >> 172.21.15.38:0/3791402528 conn(0x7f6dd40f63c0 msgr2=0x7f6dd40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:44.304 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 shutdown_connections 2023-11-15T04:08:44.304 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:44.300+0000 7f6ddadc9700 1 -- 172.21.15.38:0/3791402528 wait complete. 2023-11-15T04:08:44.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:44 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:43.294634+0000 mon.smithi038 (mon.0) 84 : audit [INF] from='client.? 172.21.15.38:0/3652390039' entity='client.admin' 2023-11-15T04:08:44.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:44 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:44.253478+0000 mon.smithi038 (mon.0) 85 : audit [INF] from='client.? 172.21.15.38:0/3791402528' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T04:08:44.727 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:08:45.189 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.188+0000 7efc78aa6700 1 Processor -- start 2023-11-15T04:08:45.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.188+0000 7efc78aa6700 1 -- start start 2023-11-15T04:08:45.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.188+0000 7efc78aa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:45.190 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.188+0000 7efc78aa6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740fec20 con 0x7efc74070430 2023-11-15T04:08:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.188+0000 7efc737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44422/0 (socket says 172.21.15.38:44422) 2023-11-15T04:08:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc737fe700 1 -- 172.21.15.38:0/605958031 learned_addr learned my addr 172.21.15.38:0/605958031 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc737fe700 1 -- 172.21.15.38:0/605958031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740fed60 con 0x7efc74070430 2023-11-15T04:08:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc737fe700 1 --2- 172.21.15.38:0/605958031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7efc6400b5f0 tx=0x7efc64006ee0).ready entity=mon.0 client_cookie=122b46af65ae9185 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc727fc700 1 -- 172.21.15.38:0/605958031 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6400d600 con 0x7efc74070430 2023-11-15T04:08:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc727fc700 1 -- 172.21.15.38:0/605958031 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc6400dbd0 con 0x7efc74070430 2023-11-15T04:08:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.189+0000 7efc727fc700 1 -- 172.21.15.38:0/605958031 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc64011c00 con 0x7efc74070430 2023-11-15T04:08:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 -- 172.21.15.38:0/605958031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 msgr2=0x7efc740fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 --2- 172.21.15.38:0/605958031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7efc6400b5f0 tx=0x7efc64006ee0).stop 2023-11-15T04:08:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 -- 172.21.15.38:0/605958031 shutdown_connections 2023-11-15T04:08:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 --2- 172.21.15.38:0/605958031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fe730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 -- 172.21.15.38:0/605958031 >> 172.21.15.38:0/605958031 conn(0x7efc740f6210 msgr2=0x7efc740f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 -- 172.21.15.38:0/605958031 shutdown_connections 2023-11-15T04:08:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.190+0000 7efc78aa6700 1 -- 172.21.15.38:0/605958031 wait complete. 2023-11-15T04:08:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc78aa6700 1 Processor -- start 2023-11-15T04:08:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc78aa6700 1 -- start start 2023-11-15T04:08:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc78aa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc78aa6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740fa630 con 0x7efc74070430 2023-11-15T04:08:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.191+0000 7efc737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44436/0 (socket says 172.21.15.38:44436) 2023-11-15T04:08:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc737fe700 1 -- 172.21.15.38:0/3638631252 learned_addr learned my addr 172.21.15.38:0/3638631252 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc737fe700 1 -- 172.21.15.38:0/3638631252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6400b200 con 0x7efc74070430 2023-11-15T04:08:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc737fe700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7efc6400e040 tx=0x7efc64006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6400d9f0 con 0x7efc74070430 2023-11-15T04:08:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc6401a9f0 con 0x7efc74070430 2023-11-15T04:08:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc740fa7d0 con 0x7efc74070430 2023-11-15T04:08:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6401b8c0 con 0x7efc74070430 2023-11-15T04:08:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.192+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc740fabe0 con 0x7efc74070430 2023-11-15T04:08:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.193+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7efc6400f070 con 0x7efc74070430 2023-11-15T04:08:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.193+0000 7efc5bfff700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 0x7efc600323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.193+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7efc64041430 con 0x7efc74070430 2023-11-15T04:08:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.193+0000 7efc72ffd700 1 -- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 msgr2=0x7efc600323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.193+0000 7efc72ffd700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 0x7efc600323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:08:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.195+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc74059100 con 0x7efc74070430 2023-11-15T04:08:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.199+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc64016070 con 0x7efc74070430 2023-11-15T04:08:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.394+0000 7efc72ffd700 1 -- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 msgr2=0x7efc600323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:45.395 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.394+0000 7efc72ffd700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 0x7efc600323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:08:45.414 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.414+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7efc74104640 con 0x7efc74070430 2023-11-15T04:08:45.415 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.414+0000 7efc5bfff700 1 -- 172.21.15.38:0/3638631252 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7efc64025460 con 0x7efc74070430 2023-11-15T04:08:45.416 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:45.416 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T04:08:45.416 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:08:45.416 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "active_name": "smithi038.bnbogz", 2023-11-15T04:08:45.416 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:08:45.417 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:45.418 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 msgr2=0x7efc600323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:08:45.418 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 0x7efc600323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:45.418 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 msgr2=0x7efc740fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7efc6400e040 tx=0x7efc64006ee0).stop 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 shutdown_connections 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7efc6002ff00 0x7efc600323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 --2- 172.21.15.38:0/3638631252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc74070430 0x7efc740fc1b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.416+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 >> 172.21.15.38:0/3638631252 conn(0x7efc740f6210 msgr2=0x7efc740f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:45.419 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.417+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 shutdown_connections 2023-11-15T04:08:45.420 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:45.417+0000 7efc78aa6700 1 -- 172.21.15.38:0/3638631252 wait complete. 2023-11-15T04:08:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:44.298482+0000 mon.smithi038 (mon.0) 86 : audit [INF] from='client.? 172.21.15.38:0/3791402528' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T04:08:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:44.298564+0000 mon.smithi038 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi038.bnbogz(active, since 7s) 2023-11-15T04:08:45.923 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for the mgr to restart... 2023-11-15T04:08:45.923 INFO:teuthology.orchestra.run.smithi038.stderr:Waiting for mgr epoch 13... 2023-11-15T04:08:45.923 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:08:46.272 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.267+0000 7f00a42e7700 1 Processor -- start 2023-11-15T04:08:46.272 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.267+0000 7f00a42e7700 1 -- start start 2023-11-15T04:08:46.272 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a42e7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:46.273 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a42e7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0fb490 con 0x7f009c0fab80 2023-11-15T04:08:46.273 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a2083700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:46.273 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a2083700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44448/0 (socket says 172.21.15.38:44448) 2023-11-15T04:08:46.273 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a2083700 1 -- 172.21.15.38:0/205945448 learned_addr learned my addr 172.21.15.38:0/205945448 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:46.274 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a2083700 1 -- 172.21.15.38:0/205945448 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0fb5d0 con 0x7f009c0fab80 2023-11-15T04:08:46.274 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.268+0000 7f00a2083700 1 --2- 172.21.15.38:0/205945448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f009800b5f0 tx=0x7f0098006ee0).ready entity=mon.0 client_cookie=854e765d2f42222a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:46.274 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a1081700 1 -- 172.21.15.38:0/205945448 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f009800d600 con 0x7f009c0fab80 2023-11-15T04:08:46.274 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a1081700 1 -- 172.21.15.38:0/205945448 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f009800dbd0 con 0x7f009c0fab80 2023-11-15T04:08:46.274 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a1081700 1 -- 172.21.15.38:0/205945448 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0098011c00 con 0x7f009c0fab80 2023-11-15T04:08:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a42e7700 1 -- 172.21.15.38:0/205945448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 msgr2=0x7f009c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a42e7700 1 --2- 172.21.15.38:0/205945448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f009800b5f0 tx=0x7f0098006ee0).stop 2023-11-15T04:08:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a42e7700 1 -- 172.21.15.38:0/205945448 shutdown_connections 2023-11-15T04:08:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a42e7700 1 --2- 172.21.15.38:0/205945448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c0fafa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.269+0000 7f00a42e7700 1 -- 172.21.15.38:0/205945448 >> 172.21.15.38:0/205945448 conn(0x7f009c0f63c0 msgr2=0x7f009c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.270+0000 7f00a42e7700 1 -- 172.21.15.38:0/205945448 shutdown_connections 2023-11-15T04:08:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.270+0000 7f00a42e7700 1 -- 172.21.15.38:0/205945448 wait complete. 2023-11-15T04:08:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.270+0000 7f00a42e7700 1 Processor -- start 2023-11-15T04:08:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.270+0000 7f00a42e7700 1 -- start start 2023-11-15T04:08:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a42e7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:46.277 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a42e7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c109e10 con 0x7f009c0fab80 2023-11-15T04:08:46.277 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a2083700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:46.277 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a2083700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44462/0 (socket says 172.21.15.38:44462) 2023-11-15T04:08:46.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a2083700 1 -- 172.21.15.38:0/1556357356 learned_addr learned my addr 172.21.15.38:0/1556357356 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:46.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a2083700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009800b200 con 0x7f009c0fab80 2023-11-15T04:08:46.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f00a2083700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f009800e040 tx=0x7f0098006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:46.279 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.271+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f009801b410 con 0x7f009c0fab80 2023-11-15T04:08:46.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.272+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f009801b9e0 con 0x7f009c0fab80 2023-11-15T04:08:46.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.272+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c10a010 con 0x7f009c0fab80 2023-11-15T04:08:46.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.272+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f009801a820 con 0x7f009c0fab80 2023-11-15T04:08:46.280 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.272+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c10a420 con 0x7f009c0fab80 2023-11-15T04:08:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.272+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f009800f070 con 0x7f009c0fab80 2023-11-15T04:08:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.273+0000 7f008affd700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.273+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f009c059100 con 0x7f008402ff00 2023-11-15T04:08:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.273+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f00980415e0 con 0x7f009c0fab80 2023-11-15T04:08:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.273+0000 7f00a1882700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:46.282 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.273+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:08:46.475 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.474+0000 7f00a1882700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:46.475 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.474+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:08:46.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:46 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:45.415260+0000 mon.smithi038 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.38:0/3638631252' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:08:46.875 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.875+0000 7f00a1882700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:46.876 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:46.875+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:08:47.676 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:47.676+0000 7f00a1882700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:47.677 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:47.676+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:08:49.277 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:49.277+0000 7f00a1882700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/27986150 2023-11-15T04:08:49.278 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:49.277+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:08:50.041 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:50.040+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f009801bb50 con 0x7f009c0fab80 2023-11-15T04:08:50.041 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:50.040+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 msgr2=0x7f00840323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:08:50.042 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:50.040+0000 7f008affd700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:50.298 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:49.988868+0000 mon.smithi038 (mon.0) 89 : cluster [INF] Active manager daemon smithi038.bnbogz restarted 2023-11-15T04:08:50.298 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:49.989621+0000 mon.smithi038 (mon.0) 90 : cluster [INF] Activating manager daemon smithi038.bnbogz 2023-11-15T04:08:50.299 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:49.993743+0000 mon.smithi038 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T04:08:51.049 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.047+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f009801a980 con 0x7f009c0fab80 2023-11-15T04:08:51.049 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.047+0000 7f008affd700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 0x7f0084035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:51.049 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.047+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f009c059100 con 0x7f00840339e0 2023-11-15T04:08:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.047+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 0x7f0084035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.048+0000 7f00a1882700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 0x7f0084035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f008c006fd0 tx=0x7f008c006d70).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.049+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f009c059100 con 0x7f00840339e0 2023-11-15T04:08:51.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f009c045ad0 con 0x7f00840339e0 2023-11-15T04:08:51.057 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f008affd700 1 -- 172.21.15.38:0/1556357356 <== mgr.14164 v2:172.21.15.38:6800/3073786548 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f009c045ad0 con 0x7f00840339e0 2023-11-15T04:08:51.059 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: { 2023-11-15T04:08:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T04:08:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:08:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: } 2023-11-15T04:08:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 msgr2=0x7f0084035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 0x7f0084035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f008c006fd0 tx=0x7f008c006d70).stop 2023-11-15T04:08:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 msgr2=0x7f009c109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f009800e040 tx=0x7f0098006ee0).stop 2023-11-15T04:08:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 shutdown_connections 2023-11-15T04:08:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f00840339e0 0x7f0084035dd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f009c0fab80 0x7f009c109920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:51.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.057+0000 7f00a42e7700 1 --2- 172.21.15.38:0/1556357356 >> [v2:172.21.15.38:6800/27986150,v1:172.21.15.38:6801/27986150] conn(0x7f008402ff00 0x7f00840323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:08:51.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.058+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 >> 172.21.15.38:0/1556357356 conn(0x7f009c0f63c0 msgr2=0x7f009c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:51.062 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.058+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 shutdown_connections 2023-11-15T04:08:51.063 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.058+0000 7f00a42e7700 1 -- 172.21.15.38:0/1556357356 wait complete. 2023-11-15T04:08:51.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:50.042628+0000 mon.smithi038 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi038.bnbogz(active, starting, since 0.0531418s) 2023-11-15T04:08:51.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.045711+0000 mon.smithi038 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata", "id": "smithi038"}]: dispatch 2023-11-15T04:08:51.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.046457+0000 mon.smithi038 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mgr metadata", "who": "smithi038.bnbogz", "id": "smithi038.bnbogz"}]: dispatch 2023-11-15T04:08:51.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.047245+0000 mon.smithi038 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:08:51.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.047428+0000 mon.smithi038 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:08:51.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.047613+0000 mon.smithi038 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:08:51.344 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:50.061560+0000 mon.smithi038 (mon.0) 98 : cluster [INF] Manager daemon smithi038.bnbogz is now available 2023-11-15T04:08:51.344 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.175775+0000 mon.smithi038 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:51.344 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.178180+0000 mon.smithi038 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:08:51.344 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.204994+0000 mon.smithi038 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:08:51.344 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:50.207924+0000 mon.smithi038 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:08:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:mgr epoch 13 is available 2023-11-15T04:08:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:Generating a dashboard self-signed certificate... 2023-11-15T04:08:51.360 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T04:08:51.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.771+0000 7f74ca051700 1 Processor -- start 2023-11-15T04:08:51.773 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.771+0000 7f74ca051700 1 -- start start 2023-11-15T04:08:51.774 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.771+0000 7f74ca051700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:51.774 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.771+0000 7f74ca051700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40fb490 con 0x7f74c40fab80 2023-11-15T04:08:51.774 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:51.774 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47498/0 (socket says 172.21.15.38:47498) 2023-11-15T04:08:51.774 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c37fe700 1 -- 172.21.15.38:0/2896418028 learned_addr learned my addr 172.21.15.38:0/2896418028 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:51.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c37fe700 1 -- 172.21.15.38:0/2896418028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40fb5d0 con 0x7f74c40fab80 2023-11-15T04:08:51.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c37fe700 1 --2- 172.21.15.38:0/2896418028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f74b400b5f0 tx=0x7f74b4006ee0).ready entity=mon.0 client_cookie=e8fba43c1dcb299 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:51.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c27fc700 1 -- 172.21.15.38:0/2896418028 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b400d600 con 0x7f74c40fab80 2023-11-15T04:08:51.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c27fc700 1 -- 172.21.15.38:0/2896418028 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f74b400dbd0 con 0x7f74c40fab80 2023-11-15T04:08:51.775 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.772+0000 7f74c27fc700 1 -- 172.21.15.38:0/2896418028 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b4011c00 con 0x7f74c40fab80 2023-11-15T04:08:51.778 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 -- 172.21.15.38:0/2896418028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 msgr2=0x7f74c40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:51.778 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 --2- 172.21.15.38:0/2896418028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f74b400b5f0 tx=0x7f74b4006ee0).stop 2023-11-15T04:08:51.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 -- 172.21.15.38:0/2896418028 shutdown_connections 2023-11-15T04:08:51.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 --2- 172.21.15.38:0/2896418028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c40fafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:51.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 -- 172.21.15.38:0/2896418028 >> 172.21.15.38:0/2896418028 conn(0x7f74c40f63c0 msgr2=0x7f74c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:51.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 -- 172.21.15.38:0/2896418028 shutdown_connections 2023-11-15T04:08:51.779 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.773+0000 7f74ca051700 1 -- 172.21.15.38:0/2896418028 wait complete. 2023-11-15T04:08:51.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74ca051700 1 Processor -- start 2023-11-15T04:08:51.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74ca051700 1 -- start start 2023-11-15T04:08:51.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74ca051700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:51.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74ca051700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c410af10 con 0x7f74c40fab80 2023-11-15T04:08:51.780 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:51.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47502/0 (socket says 172.21.15.38:47502) 2023-11-15T04:08:51.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.774+0000 7f74c37fe700 1 -- 172.21.15.38:0/2129502353 learned_addr learned my addr 172.21.15.38:0/2129502353 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:51.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74c37fe700 1 -- 172.21.15.38:0/2129502353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b400b200 con 0x7f74c40fab80 2023-11-15T04:08:51.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74c37fe700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f74b400e040 tx=0x7f74b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:51.781 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b401b410 con 0x7f74c40fab80 2023-11-15T04:08:51.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c410b110 con 0x7f74c40fab80 2023-11-15T04:08:51.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f74b401b9e0 con 0x7f74c40fab80 2023-11-15T04:08:51.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c410b520 con 0x7f74c40fab80 2023-11-15T04:08:51.782 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.775+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b401a820 con 0x7f74c40fab80 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.776+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f74b400f070 con 0x7f74c40fab80 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.776+0000 7f74abfff700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 0x7f74b0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.776+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74b4042250 con 0x7f74c40fab80 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.776+0000 7f74c2ffd700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 0x7f74b0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.776+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c4059100 con 0x7f74c40fab80 2023-11-15T04:08:51.783 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.777+0000 7f74c2ffd700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 0x7f74b0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f74b80096a0 tx=0x7f74b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:51.784 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.781+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74b4016070 con 0x7f74c40fab80 2023-11-15T04:08:51.955 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:51.954+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c41049a0 con 0x7f74b002fd50 2023-11-15T04:08:52.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.015+0000 7f74abfff700 1 -- 172.21.15.38:0/2129502353 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f74c41049a0 con 0x7f74b002fd50 2023-11-15T04:08:52.016 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T04:08:52.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 msgr2=0x7f74b0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:52.018 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 0x7f74b0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f74b80096a0 tx=0x7f74b8006b40).stop 2023-11-15T04:08:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 msgr2=0x7f74c410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f74b400e040 tx=0x7f74b4006ee0).stop 2023-11-15T04:08:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 shutdown_connections 2023-11-15T04:08:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f74b002fd50 0x7f74b0032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 --2- 172.21.15.38:0/2129502353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c40fab80 0x7f74c410aa20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:52.020 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 >> 172.21.15.38:0/2129502353 conn(0x7f74c40f63c0 msgr2=0x7f74c4070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:52.020 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 shutdown_connections 2023-11-15T04:08:52.020 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.017+0000 7f74ca051700 1 -- 172.21.15.38:0/2129502353 wait complete. 2023-11-15T04:08:52.314 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:51.049850+0000 mgr.smithi038.bnbogz (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:08:52.315 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:51.050746+0000 mon.smithi038 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi038.bnbogz(active, since 1.06126s) 2023-11-15T04:08:52.315 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:51.058184+0000 mgr.smithi038.bnbogz (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:08:52.315 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:52.011306+0000 mon.smithi038 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:52.315 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:52.015798+0000 mon.smithi038 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:52.386 INFO:teuthology.orchestra.run.smithi038.stderr:Creating initial admin user... 2023-11-15T04:08:52.387 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp81fr4k36:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T04:08:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.747+0000 7f708ca44700 1 Processor -- start 2023-11-15T04:08:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.747+0000 7f708ca44700 1 -- start start 2023-11-15T04:08:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708ca44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708ca44700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880fc750 con 0x7f70880703b0 2023-11-15T04:08:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47504/0 (socket says 172.21.15.38:47504) 2023-11-15T04:08:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708659c700 1 -- 172.21.15.38:0/2626437758 learned_addr learned my addr 172.21.15.38:0/2626437758 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708659c700 1 -- 172.21.15.38:0/2626437758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088101250 con 0x7f70880703b0 2023-11-15T04:08:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708659c700 1 --2- 172.21.15.38:0/2626437758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f707000b5f0 tx=0x7f7070006ee0).ready entity=mon.0 client_cookie=cb623d928feb24f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708559a700 1 -- 172.21.15.38:0/2626437758 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707000d600 con 0x7f70880703b0 2023-11-15T04:08:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708559a700 1 -- 172.21.15.38:0/2626437758 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f707000dbd0 con 0x7f70880703b0 2023-11-15T04:08:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.748+0000 7f708559a700 1 -- 172.21.15.38:0/2626437758 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7070011c00 con 0x7f70880703b0 2023-11-15T04:08:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 -- 172.21.15.38:0/2626437758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 msgr2=0x7f7088100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 --2- 172.21.15.38:0/2626437758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f707000b5f0 tx=0x7f7070006ee0).stop 2023-11-15T04:08:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 -- 172.21.15.38:0/2626437758 shutdown_connections 2023-11-15T04:08:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 --2- 172.21.15.38:0/2626437758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f7088100d60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 -- 172.21.15.38:0/2626437758 >> 172.21.15.38:0/2626437758 conn(0x7f70880f63c0 msgr2=0x7f70880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 -- 172.21.15.38:0/2626437758 shutdown_connections 2023-11-15T04:08:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.749+0000 7f708ca44700 1 -- 172.21.15.38:0/2626437758 wait complete. 2023-11-15T04:08:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.750+0000 7f708ca44700 1 Processor -- start 2023-11-15T04:08:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.750+0000 7f708ca44700 1 -- start start 2023-11-15T04:08:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.750+0000 7f708ca44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:52.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.750+0000 7f708ca44700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708810d100 con 0x7f70880703b0 2023-11-15T04:08:52.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:52.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47518/0 (socket says 172.21.15.38:47518) 2023-11-15T04:08:52.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708659c700 1 -- 172.21.15.38:0/197514262 learned_addr learned my addr 172.21.15.38:0/197514262 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:52.756 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708659c700 1 -- 172.21.15.38:0/197514262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707000b200 con 0x7f70880703b0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708659c700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f707000b190 tx=0x7f7070006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707001b410 con 0x7f70880703b0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f707001b9e0 con 0x7f70880703b0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707001a880 con 0x7f70880703b0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708810d300 con 0x7f70880703b0 2023-11-15T04:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.751+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708810d710 con 0x7f70880703b0 2023-11-15T04:08:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.752+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f707000f070 con 0x7f70880703b0 2023-11-15T04:08:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.752+0000 7f707effd700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 0x7f7068032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.752+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70700414f0 con 0x7f70880703b0 2023-11-15T04:08:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.752+0000 7f7085d9b700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 0x7f7068032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.753+0000 7f7085d9b700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 0x7f7068032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.753+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7088059100 con 0x7f70880703b0 2023-11-15T04:08:52.760 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.758+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7070022d50 con 0x7f70880703b0 2023-11-15T04:08:52.943 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:52.943+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7088045ad0 con 0x7f706802feb0 2023-11-15T04:08:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.198+0000 7f707effd700 1 -- 172.21.15.38:0/197514262 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7088045ad0 con 0x7f706802feb0 2023-11-15T04:08:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$GjItRr71jr07fvx8bsmSo.E/7ho/h1FQ.0jj86FD.ymxGwWw1GVCu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700021333, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T04:08:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 msgr2=0x7f7068032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 0x7f7068032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).stop 2023-11-15T04:08:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 msgr2=0x7f708810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f707000b190 tx=0x7f7070006ee0).stop 2023-11-15T04:08:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 shutdown_connections 2023-11-15T04:08:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f706802feb0 0x7f7068032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 --2- 172.21.15.38:0/197514262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70880703b0 0x7f708810cc10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.200+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 >> 172.21.15.38:0/197514262 conn(0x7f70880f63c0 msgr2=0x7f70880f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.201+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 shutdown_connections 2023-11-15T04:08:53.203 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.201+0000 7f708ca44700 1 -- 172.21.15.38:0/197514262 wait complete. 2023-11-15T04:08:53.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:51.955897+0000 mgr.smithi038.bnbogz (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:53.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:52.395487+0000 mon.smithi038 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi038.bnbogz(active, since 2s) 2023-11-15T04:08:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:Fetching dashboard port number... 2023-11-15T04:08:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpy0mcl6p9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpbf936z68:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T04:08:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.985+0000 7f7a57361700 1 Processor -- start 2023-11-15T04:08:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.985+0000 7f7a57361700 1 -- start start 2023-11-15T04:08:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.985+0000 7f7a57361700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.985+0000 7f7a57361700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500fc730 con 0x7f7a500703d0 2023-11-15T04:08:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.985+0000 7f7a550fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a550fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47522/0 (socket says 172.21.15.38:47522) 2023-11-15T04:08:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a550fd700 1 -- 172.21.15.38:0/3247075087 learned_addr learned my addr 172.21.15.38:0/3247075087 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a550fd700 1 -- 172.21.15.38:0/3247075087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50101230 con 0x7f7a500703d0 2023-11-15T04:08:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a550fd700 1 --2- 172.21.15.38:0/3247075087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7a4c00b5f0 tx=0x7f7a4c006ee0).ready entity=mon.0 client_cookie=fcad9bff6106abad server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a3ffff700 1 -- 172.21.15.38:0/3247075087 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c00d600 con 0x7f7a500703d0 2023-11-15T04:08:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a3ffff700 1 -- 172.21.15.38:0/3247075087 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7a4c00dbd0 con 0x7f7a500703d0 2023-11-15T04:08:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.986+0000 7f7a3ffff700 1 -- 172.21.15.38:0/3247075087 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c011c00 con 0x7f7a500703d0 2023-11-15T04:08:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 -- 172.21.15.38:0/3247075087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 msgr2=0x7f7a50100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 --2- 172.21.15.38:0/3247075087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7a4c00b5f0 tx=0x7f7a4c006ee0).stop 2023-11-15T04:08:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 -- 172.21.15.38:0/3247075087 shutdown_connections 2023-11-15T04:08:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 --2- 172.21.15.38:0/3247075087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a50100d40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 -- 172.21.15.38:0/3247075087 >> 172.21.15.38:0/3247075087 conn(0x7f7a500f63c0 msgr2=0x7f7a500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 -- 172.21.15.38:0/3247075087 shutdown_connections 2023-11-15T04:08:53.992 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.987+0000 7f7a57361700 1 -- 172.21.15.38:0/3247075087 wait complete. 2023-11-15T04:08:53.992 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a57361700 1 Processor -- start 2023-11-15T04:08:53.992 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a57361700 1 -- start start 2023-11-15T04:08:53.993 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a57361700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:53.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a57361700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5010d0a0 con 0x7f7a500703d0 2023-11-15T04:08:53.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a550fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:53.994 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a550fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47538/0 (socket says 172.21.15.38:47538) 2023-11-15T04:08:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a550fd700 1 -- 172.21.15.38:0/80068029 learned_addr learned my addr 172.21.15.38:0/80068029 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.988+0000 7f7a550fd700 1 -- 172.21.15.38:0/80068029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c00b200 con 0x7f7a500703d0 2023-11-15T04:08:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a550fd700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7a4c00b190 tx=0x7f7a4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c01b410 con 0x7f7a500703d0 2023-11-15T04:08:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5010d2a0 con 0x7f7a500703d0 2023-11-15T04:08:53.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7a4c01b9e0 con 0x7f7a500703d0 2023-11-15T04:08:53.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c01a880 con 0x7f7a500703d0 2023-11-15T04:08:53.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.989+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5010d6b0 con 0x7f7a500703d0 2023-11-15T04:08:53.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.990+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50045ad0 con 0x7f7a500703d0 2023-11-15T04:08:53.996 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.990+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f7a4c00f070 con 0x7f7a500703d0 2023-11-15T04:08:53.997 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.990+0000 7f7a3dffb700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 0x7f7a380323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:53.997 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.991+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a4c041430 con 0x7f7a500703d0 2023-11-15T04:08:53.997 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.991+0000 7f7a548fc700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 0x7f7a380323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:53.997 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.991+0000 7f7a548fc700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 0x7f7a380323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7a400096a0 tx=0x7f7a40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:53.998 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:53.995+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a4c019950 con 0x7f7a500703d0 2023-11-15T04:08:54.163 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.162+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7a50101d90 con 0x7f7a500703d0 2023-11-15T04:08:54.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.163+0000 7f7a3dffb700 1 -- 172.21.15.38:0/80068029 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7a4c016020 con 0x7f7a500703d0 2023-11-15T04:08:54.164 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 8443 2023-11-15T04:08:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.164+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 msgr2=0x7f7a380323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 0x7f7a380323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7a400096a0 tx=0x7f7a40006b40).stop 2023-11-15T04:08:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 msgr2=0x7f7a5010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7a4c00b190 tx=0x7f7a4c006ee0).stop 2023-11-15T04:08:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 shutdown_connections 2023-11-15T04:08:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a3802ff00 0x7f7a380323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 --2- 172.21.15.38:0/80068029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a500703d0 0x7f7a5010cbb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:54.167 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 >> 172.21.15.38:0/80068029 conn(0x7f7a500f63c0 msgr2=0x7f7a500f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 shutdown_connections 2023-11-15T04:08:54.168 INFO:teuthology.orchestra.run.smithi038.stderr:/usr/bin/ceph: 2023-11-15T04:08:54.165+0000 7f7a57361700 1 -- 172.21.15.38:0/80068029 wait complete. 2023-11-15T04:08:54.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:52.944872+0000 mgr.smithi038.bnbogz (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:08:54.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:53.198780+0000 mon.smithi038 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:54.506 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:54.163976+0000 mon.smithi038 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.38:0/80068029' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T04:08:54.562 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:08:54.573 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: disabled 2023-11-15T04:08:54.574 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:08:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:systemctl: inactive 2023-11-15T04:08:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:firewalld.service is not enabled 2023-11-15T04:08:54.584 INFO:teuthology.orchestra.run.smithi038.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T04:08:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:Ceph Dashboard is now available at: 2023-11-15T04:08:54.587 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.588 INFO:teuthology.orchestra.run.smithi038.stderr: URL: https://smithi038.front.sepia.ceph.com:8443/ 2023-11-15T04:08:54.588 INFO:teuthology.orchestra.run.smithi038.stderr: User: admin 2023-11-15T04:08:54.588 INFO:teuthology.orchestra.run.smithi038.stderr: Password: w6h7p7ovmb 2023-11-15T04:08:54.588 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:You can access the Ceph CLI with: 2023-11-15T04:08:54.589 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.590 INFO:teuthology.orchestra.run.smithi038.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:08:54.590 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.590 INFO:teuthology.orchestra.run.smithi038.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T04:08:54.591 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.591 INFO:teuthology.orchestra.run.smithi038.stderr: ceph telemetry on 2023-11-15T04:08:54.591 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:For more information see: 2023-11-15T04:08:54.592 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.592 INFO:teuthology.orchestra.run.smithi038.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T04:08:54.593 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:08:54.593 INFO:teuthology.orchestra.run.smithi038.stderr:Bootstrap complete. 2023-11-15T04:08:54.593 INFO:teuthology.orchestra.run.smithi038.stderr:Releasing lock 140269897492968 on /run/cephadm/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:08:54.593 INFO:teuthology.orchestra.run.smithi038.stderr:Lock 140269897492968 released on /run/cephadm/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:08:54.629 INFO:tasks.cephadm:Fetching config... 2023-11-15T04:08:54.630 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:08:54.630 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T04:08:54.692 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T04:08:54.692 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:08:54.693 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T04:08:54.754 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T04:08:54.755 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:08:54.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/mon.smithi038/keyring of=/dev/stdout 2023-11-15T04:08:54.830 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T04:08:54.830 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-15T04:08:54.830 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T04:08:54.894 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T04:08:54.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAPzcLBh8oQd+ZPKGSU3Yqs8C4Iv0wSo2Z9jWPNZmoau26qVWs9/8LlVeCIHNaMtyGVEJokXXv+6VtGYU7Te9LiW//mO5pqFD2PErjjERHWggbUDSJ0z6ty8y/xPoEgmy65FPZ7JNwLX4wgdez7eOkbU2Z008MPdNktY4iOs793Vvkx1qNVQi8tWwI3Rqcz/r5GyU4qrQ2P+XzjvtKlDaY6ZN63GkFF/rviS9EWIBDJcZcSTB9jrOnua+ufU2Ootx/BA3aD9Usze5hE7bwoZnatLknFVp0wdJ3vwkEL5Cwho4M3CltPMuNJEANmTh54mhbdqLmLQLU52rEe1zrSA3In8rLN4TpuGT3KKtWAYTze7lQtQ/zLE0sh1pq+cLIA1eY/4WgPv6RGfOfi5EonoxJzt7+mem5rWVNzt0YJvAegzXobca2Znyoc8qhjgBCV87uVIcyMH8X77fxQhEWp2qEOLfp3JX/Uyt6lj0oV0THbMLBIHW38gkzAxvz2jcqUU0= ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAPzcLBh8oQd+ZPKGSU3Yqs8C4Iv0wSo2Z9jWPNZmoau26qVWs9/8LlVeCIHNaMtyGVEJokXXv+6VtGYU7Te9LiW//mO5pqFD2PErjjERHWggbUDSJ0z6ty8y/xPoEgmy65FPZ7JNwLX4wgdez7eOkbU2Z008MPdNktY4iOs793Vvkx1qNVQi8tWwI3Rqcz/r5GyU4qrQ2P+XzjvtKlDaY6ZN63GkFF/rviS9EWIBDJcZcSTB9jrOnua+ufU2Ootx/BA3aD9Usze5hE7bwoZnatLknFVp0wdJ3vwkEL5Cwho4M3CltPMuNJEANmTh54mhbdqLmLQLU52rEe1zrSA3In8rLN4TpuGT3KKtWAYTze7lQtQ/zLE0sh1pq+cLIA1eY/4WgPv6RGfOfi5EonoxJzt7+mem5rWVNzt0YJvAegzXobca2Znyoc8qhjgBCV87uVIcyMH8X77fxQhEWp2qEOLfp3JX/Uyt6lj0oV0THbMLBIHW38gkzAxvz2jcqUU0= ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:08:54.993 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAPzcLBh8oQd+ZPKGSU3Yqs8C4Iv0wSo2Z9jWPNZmoau26qVWs9/8LlVeCIHNaMtyGVEJokXXv+6VtGYU7Te9LiW//mO5pqFD2PErjjERHWggbUDSJ0z6ty8y/xPoEgmy65FPZ7JNwLX4wgdez7eOkbU2Z008MPdNktY4iOs793Vvkx1qNVQi8tWwI3Rqcz/r5GyU4qrQ2P+XzjvtKlDaY6ZN63GkFF/rviS9EWIBDJcZcSTB9jrOnua+ufU2Ootx/BA3aD9Usze5hE7bwoZnatLknFVp0wdJ3vwkEL5Cwho4M3CltPMuNJEANmTh54mhbdqLmLQLU52rEe1zrSA3In8rLN4TpuGT3KKtWAYTze7lQtQ/zLE0sh1pq+cLIA1eY/4WgPv6RGfOfi5EonoxJzt7+mem5rWVNzt0YJvAegzXobca2Znyoc8qhjgBCV87uVIcyMH8X77fxQhEWp2qEOLfp3JX/Uyt6lj0oV0THbMLBIHW38gkzAxvz2jcqUU0= ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:08:55.037 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAPzcLBh8oQd+ZPKGSU3Yqs8C4Iv0wSo2Z9jWPNZmoau26qVWs9/8LlVeCIHNaMtyGVEJokXXv+6VtGYU7Te9LiW//mO5pqFD2PErjjERHWggbUDSJ0z6ty8y/xPoEgmy65FPZ7JNwLX4wgdez7eOkbU2Z008MPdNktY4iOs793Vvkx1qNVQi8tWwI3Rqcz/r5GyU4qrQ2P+XzjvtKlDaY6ZN63GkFF/rviS9EWIBDJcZcSTB9jrOnua+ufU2Ootx/BA3aD9Usze5hE7bwoZnatLknFVp0wdJ3vwkEL5Cwho4M3CltPMuNJEANmTh54mhbdqLmLQLU52rEe1zrSA3In8rLN4TpuGT3KKtWAYTze7lQtQ/zLE0sh1pq+cLIA1eY/4WgPv6RGfOfi5EonoxJzt7+mem5rWVNzt0YJvAegzXobca2Znyoc8qhjgBCV87uVIcyMH8X77fxQhEWp2qEOLfp3JX/Uyt6lj0oV0THbMLBIHW38gkzAxvz2jcqUU0= ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 2023-11-15T04:08:55.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T04:08:56.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 -- 172.21.15.38:0/1578182133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 msgr2=0x7f05280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:56.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 --2- 172.21.15.38:0/1578182133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05280f66c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0518009fb0 tx=0x7f0518009b30).stop 2023-11-15T04:08:56.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 -- 172.21.15.38:0/1578182133 shutdown_connections 2023-11-15T04:08:56.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 --2- 172.21.15.38:0/1578182133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05280f66c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:56.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 -- 172.21.15.38:0/1578182133 >> 172.21.15.38:0/1578182133 conn(0x7f05280f1aa0 msgr2=0x7f05280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:56.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 -- 172.21.15.38:0/1578182133 shutdown_connections 2023-11-15T04:08:56.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.606+0000 7f052c929700 1 -- 172.21.15.38:0/1578182133 wait complete. 2023-11-15T04:08:56.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f052c929700 1 Processor -- start 2023-11-15T04:08:56.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f052c929700 1 -- start start 2023-11-15T04:08:56.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f052c929700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:56.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f052c929700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05281069b0 con 0x7f05280f62a0 2023-11-15T04:08:56.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f0526d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:56.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f0526d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47560/0 (socket says 172.21.15.38:47560) 2023-11-15T04:08:56.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f0526d9d700 1 -- 172.21.15.38:0/2448445032 learned_addr learned my addr 172.21.15.38:0/2448445032 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:08:56.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.607+0000 7f0526d9d700 1 -- 172.21.15.38:0/2448445032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518005040 con 0x7f05280f62a0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f0526d9d700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0518002040 tx=0x7f051800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05180095f0 con 0x7f05280f62a0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0518007e30 con 0x7f05280f62a0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0528106bb0 con 0x7f05280f62a0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0518014d20 con 0x7f05280f62a0 2023-11-15T04:08:56.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.608+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0528107040 con 0x7f05280f62a0 2023-11-15T04:08:56.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0518025070 con 0x7f05280f62a0 2023-11-15T04:08:56.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f050f7fe700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 0x7f0510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:08:56.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0518040aa0 con 0x7f05280f62a0 2023-11-15T04:08:56.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f052659c700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 0x7f0510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:08:56.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0528059100 con 0x7f05280f62a0 2023-11-15T04:08:56.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.609+0000 7f052659c700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 0x7f0510032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f051c0096a0 tx=0x7f051c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:08:56.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.614+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f051800f300 con 0x7f05280f62a0 2023-11-15T04:08:56.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.777+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0528045ad0 con 0x7f05280f62a0 2023-11-15T04:08:56.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.783+0000 7f050f7fe700 1 -- 172.21.15.38:0/2448445032 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0518025ab0 con 0x7f05280f62a0 2023-11-15T04:08:56.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 msgr2=0x7f0510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:56.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 0x7f0510032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f051c0096a0 tx=0x7f051c006b40).stop 2023-11-15T04:08:56.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 msgr2=0x7f05281064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:08:56.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0518002040 tx=0x7f051800b040).stop 2023-11-15T04:08:56.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 shutdown_connections 2023-11-15T04:08:56.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f051002ff60 0x7f0510032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:56.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 --2- 172.21.15.38:0/2448445032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05280f62a0 0x7f05281064c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:08:56.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 >> 172.21.15.38:0/2448445032 conn(0x7f05280f1aa0 msgr2=0x7f05280f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:08:56.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 shutdown_connections 2023-11-15T04:08:56.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:08:56.785+0000 7f052c929700 1 -- 172.21.15.38:0/2448445032 wait complete. 2023-11-15T04:08:57.111 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2023-11-15T04:08:57.112 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:08:57.112 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2023-11-15T04:08:57.134 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-11-15T04:08:57.134 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T04:08:57.196 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2023-11-15T04:08:57.196 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi188 2023-11-15T04:08:57.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:08:56.082015+0000 mon.smithi038 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi038.bnbogz(active, since 6s) 2023-11-15T04:08:57.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:56.784201+0000 mon.smithi038 (mon.0) 110 : audit [INF] from='client.? 172.21.15.38:0/2448445032' entity='client.admin' 2023-11-15T04:08:59.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:58.176051+0000 mon.smithi038 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:59.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:58.428344+0000 mon.smithi038 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:08:59.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:08:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:08:59.160513+0000 mon.smithi038 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:03.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:01.699793+0000 mon.smithi038 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:03.093 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:02.004798+0000 mon.smithi038 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:03.093 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:02.014683+0000 mon.smithi038 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:03.093 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:09:02.021284+0000 mgr.smithi038.bnbogz (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi038 on smithi038 2023-11-15T04:09:11.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:10.048720+0000 mgr.smithi038.bnbogz (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:12.049041+0000 mgr.smithi038.bnbogz (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:14.049369+0000 mgr.smithi038.bnbogz (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:16.049645+0000 mgr.smithi038.bnbogz (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:18.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/1244277490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 msgr2=0x7ff0f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:18.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/1244277490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f40f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff0e4009fb0 tx=0x7ff0e4009b30).stop 2023-11-15T04:09:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/1244277490 shutdown_connections 2023-11-15T04:09:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/1244277490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f40f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/1244277490 >> 172.21.15.188:0/1244277490 conn(0x7ff0f40f0b70 msgr2=0x7ff0f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/1244277490 shutdown_connections 2023-11-15T04:09:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.909+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/1244277490 wait complete. 2023-11-15T04:09:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0fb8cd700 1 Processor -- start 2023-11-15T04:09:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0fb8cd700 1 -- start start 2023-11-15T04:09:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0fb8cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0fb8cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f4106c20 con 0x7ff0f40f5410 2023-11-15T04:09:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0f9669700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0f9669700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50244/0 (socket says 172.21.15.188:50244) 2023-11-15T04:09:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0f9669700 1 -- 172.21.15.188:0/2137763989 learned_addr learned my addr 172.21.15.188:0/2137763989 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.910+0000 7ff0f9669700 1 -- 172.21.15.188:0/2137763989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4005040 con 0x7ff0f40f5410 2023-11-15T04:09:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0f9669700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff0e4013510 tx=0x7ff0e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:18.919 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e4009320 con 0x7ff0f40f5410 2023-11-15T04:09:18.919 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f4106e20 con 0x7ff0f40f5410 2023-11-15T04:09:18.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0e4013f00 con 0x7ff0f40f5410 2023-11-15T04:09:18.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e4014d90 con 0x7ff0f40f5410 2023-11-15T04:09:18.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.911+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f41072b0 con 0x7ff0f40f5410 2023-11-15T04:09:18.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.912+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0e40263a0 con 0x7ff0f40f5410 2023-11-15T04:09:18.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.912+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f4045ad0 con 0x7ff0f40f5410 2023-11-15T04:09:18.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.912+0000 7ff0e9ffb700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 0x7ff0e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:18.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.912+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0e4040fe0 con 0x7ff0f40f5410 2023-11-15T04:09:18.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.912+0000 7ff0f8e68700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 0x7ff0e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:18.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.913+0000 7ff0f8e68700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 0x7ff0e0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:18.922 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:18.918+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0e40126f0 con 0x7ff0f40f5410 2023-11-15T04:09:19.088 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.088+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0f40f1d70 con 0x7ff0e002ff60 2023-11-15T04:09:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:18.049989+0000 mgr.smithi038.bnbogz (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:19.700 INFO:teuthology.orchestra.run.smithi188.stdout:Added host 'smithi188' 2023-11-15T04:09:19.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.700+0000 7ff0e9ffb700 1 -- 172.21.15.188:0/2137763989 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7ff0f40f1d70 con 0x7ff0e002ff60 2023-11-15T04:09:19.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 msgr2=0x7ff0e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:19.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 0x7ff0e0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).stop 2023-11-15T04:09:19.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 msgr2=0x7ff0f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:19.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff0e4013510 tx=0x7ff0e400b040).stop 2023-11-15T04:09:19.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 shutdown_connections 2023-11-15T04:09:19.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff0e002ff60 0x7ff0e0032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:19.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 --2- 172.21.15.188:0/2137763989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0f40f5410 0x7ff0f4106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:19.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 >> 172.21.15.188:0/2137763989 conn(0x7ff0f40f0b70 msgr2=0x7ff0f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:19.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 shutdown_connections 2023-11-15T04:09:19.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:19.702+0000 7ff0fb8cd700 1 -- 172.21.15.188:0/2137763989 wait complete. 2023-11-15T04:09:20.022 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T04:09:20.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:20 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:19.089647+0000 mgr.smithi038.bnbogz (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:09:20.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:20 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:19.699675+0000 mon.smithi038 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:21.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 -- 172.21.15.188:0/1390902710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 msgr2=0x7f19bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:21.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 --2- 172.21.15.188:0/1390902710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc0f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f19b0009fb0 tx=0x7f19b0009b30).stop 2023-11-15T04:09:21.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 -- 172.21.15.188:0/1390902710 shutdown_connections 2023-11-15T04:09:21.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 --2- 172.21.15.188:0/1390902710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc0f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:21.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 -- 172.21.15.188:0/1390902710 >> 172.21.15.188:0/1390902710 conn(0x7f19bc0f09a0 msgr2=0x7f19bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:21.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 -- 172.21.15.188:0/1390902710 shutdown_connections 2023-11-15T04:09:21.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 -- 172.21.15.188:0/1390902710 wait complete. 2023-11-15T04:09:21.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.140+0000 7f19c120d700 1 Processor -- start 2023-11-15T04:09:21.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19c120d700 1 -- start start 2023-11-15T04:09:21.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19c120d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:21.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19c120d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc1069b0 con 0x7f19bc0f51a0 2023-11-15T04:09:21.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19bbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:21.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19bbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50266/0 (socket says 172.21.15.188:50266) 2023-11-15T04:09:21.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.141+0000 7f19bbfff700 1 -- 172.21.15.188:0/1617098901 learned_addr learned my addr 172.21.15.188:0/1617098901 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:21.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19bbfff700 1 -- 172.21.15.188:0/1617098901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0005040 con 0x7f19bc0f51a0 2023-11-15T04:09:21.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19bbfff700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f19b0009830 tx=0x7f19b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:21.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b0009320 con 0x7f19bc0f51a0 2023-11-15T04:09:21.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc106bb0 con 0x7f19bc0f51a0 2023-11-15T04:09:21.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b00136c0 con 0x7f19bc0f51a0 2023-11-15T04:09:21.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b001ee90 con 0x7f19bc0f51a0 2023-11-15T04:09:21.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.142+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc107040 con 0x7f19bc0f51a0 2023-11-15T04:09:21.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.143+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19b001e420 con 0x7f19bc0f51a0 2023-11-15T04:09:21.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.143+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc045ad0 con 0x7f19bc0f51a0 2023-11-15T04:09:21.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.143+0000 7f19b8ff9700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 0x7f19a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:21.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.143+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19b0040fe0 con 0x7f19bc0f51a0 2023-11-15T04:09:21.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.143+0000 7f19bb7fe700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 0x7f19a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:21.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.144+0000 7f19bb7fe700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 0x7f19a40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f19ac0096a0 tx=0x7f19ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:21.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.148+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19b001c790 con 0x7f19bc0f51a0 2023-11-15T04:09:21.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.315+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19bc0f1b70 con 0x7f19a402ff10 2023-11-15T04:09:21.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.317+0000 7f19b8ff9700 1 -- 172.21.15.188:0/1617098901 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f19bc0f1b70 con 0x7f19a402ff10 2023-11-15T04:09:21.317 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:21.317 INFO:teuthology.orchestra.run.smithi188.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "smithi188", "hostname": "smithi188", "labels": [], "status": ""}] 2023-11-15T04:09:21.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 msgr2=0x7f19a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:21.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 0x7f19a40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f19ac0096a0 tx=0x7f19ac006b40).stop 2023-11-15T04:09:21.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 msgr2=0x7f19bc1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:21.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f19b0009830 tx=0x7f19b000b040).stop 2023-11-15T04:09:21.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 shutdown_connections 2023-11-15T04:09:21.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f19a402ff10 0x7f19a40323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:21.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 --2- 172.21.15.188:0/1617098901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19bc0f51a0 0x7f19bc1064c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:21.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 >> 172.21.15.188:0/1617098901 conn(0x7f19bc0f09a0 msgr2=0x7f19bc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:21.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 shutdown_connections 2023-11-15T04:09:21.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:21.319+0000 7f19c120d700 1 -- 172.21.15.188:0/1617098901 wait complete. 2023-11-15T04:09:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:09:19.700166+0000 mgr.smithi038.bnbogz (mgr.14164) 12 : cephadm [INF] Added host smithi188 2023-11-15T04:09:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:20.050309+0000 mgr.smithi038.bnbogz (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:21.699 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T04:09:21.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T04:09:22.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:21.316959+0000 mgr.smithi038.bnbogz (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T04:09:23.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 -- 172.21.15.38:0/378855992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 msgr2=0x7f0c0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:23.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/378855992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0c08009fb0 tx=0x7f0c08009b30).stop 2023-11-15T04:09:23.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 -- 172.21.15.38:0/378855992 shutdown_connections 2023-11-15T04:09:23.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/378855992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:23.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 -- 172.21.15.38:0/378855992 >> 172.21.15.38:0/378855992 conn(0x7f0c0c0f09a0 msgr2=0x7f0c0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:23.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.072+0000 7f0c12d9f700 1 -- 172.21.15.38:0/378855992 shutdown_connections 2023-11-15T04:09:23.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c12d9f700 1 -- 172.21.15.38:0/378855992 wait complete. 2023-11-15T04:09:23.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c12d9f700 1 Processor -- start 2023-11-15T04:09:23.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c12d9f700 1 -- start start 2023-11-15T04:09:23.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c12d9f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:23.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c12d9f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0fb5a0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c11d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:23.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c11d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42458/0 (socket says 172.21.15.38:42458) 2023-11-15T04:09:23.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.073+0000 7f0c11d9d700 1 -- 172.21.15.38:0/215244505 learned_addr learned my addr 172.21.15.38:0/215244505 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-11-15T04:09:23.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c11d9d700 1 -- 172.21.15.38:0/215244505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08005040 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c11d9d700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0c08013510 tx=0x7f0c0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:23.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c08009320 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c0fb740 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c08013f00 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c08014dc0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.074+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c0fbbd0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.075+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c080095f0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.075+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c059100 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.075+0000 7f0c027fc700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 0x7f0bf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:23.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.075+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c080410c0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.075+0000 7f0c1159c700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 0x7f0bf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:23.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.076+0000 7f0c1159c700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 0x7f0bf8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:23.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.080+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c08014420 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:23.252+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0c0c0fabe0 con 0x7f0c0c0f51a0 2023-11-15T04:09:23.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:22.050548+0000 mgr.smithi038.bnbogz (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:24.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.139+0000 7f0c027fc700 1 -- 172.21.15.38:0/215244505 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0c08010230 con 0x7f0c0c0f51a0 2023-11-15T04:09:24.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 msgr2=0x7f0bf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:24.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 0x7f0bf8032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).stop 2023-11-15T04:09:24.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 msgr2=0x7f0c0c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:24.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0c08013510 tx=0x7f0c0800b040).stop 2023-11-15T04:09:24.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 shutdown_connections 2023-11-15T04:09:24.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0bf802ff60 0x7f0bf8032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:24.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 --2- 172.21.15.38:0/215244505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0fd120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:24.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.141+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 >> 172.21.15.38:0/215244505 conn(0x7f0c0c0f09a0 msgr2=0x7f0c0c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:24.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.142+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 shutdown_connections 2023-11-15T04:09:24.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-15T04:09:24.142+0000 7f0c12d9f700 1 -- 172.21.15.38:0/215244505 wait complete. 2023-11-15T04:09:24.144 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2023-11-15T04:09:24.468 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:24 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:23.253190+0000 mon.smithi038 (mon.0) 118 : audit [INF] from='client.? 172.21.15.38:0/215244505' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T04:09:24.534 INFO:tasks.cephadm:Adding mon.smithi038 on smithi038 2023-11-15T04:09:24.535 INFO:tasks.cephadm:Adding mon.smithi188 on smithi188 2023-11-15T04:09:24.535 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi038:172.21.15.38=smithi038;smithi188:172.21.15.188=smithi188' 2023-11-15T04:09:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:24.050953+0000 mgr.smithi038.bnbogz (mgr.14164) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:24.140256+0000 mon.smithi038 (mon.0) 119 : audit [INF] from='client.? 172.21.15.38:0/215244505' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T04:09:25.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:24.140312+0000 mon.smithi038 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T04:09:25.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.736+0000 7f9d6988b700 1 -- 172.21.15.188:0/3316587998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 msgr2=0x7f9d640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:25.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.736+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3316587998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9d58009fb0 tx=0x7f9d58009b30).stop 2023-11-15T04:09:25.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.736+0000 7f9d6988b700 1 -- 172.21.15.188:0/3316587998 shutdown_connections 2023-11-15T04:09:25.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.736+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3316587998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:25.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.736+0000 7f9d6988b700 1 -- 172.21.15.188:0/3316587998 >> 172.21.15.188:0/3316587998 conn(0x7f9d640f0b70 msgr2=0x7f9d640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:25.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 -- 172.21.15.188:0/3316587998 shutdown_connections 2023-11-15T04:09:25.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 -- 172.21.15.188:0/3316587998 wait complete. 2023-11-15T04:09:25.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 Processor -- start 2023-11-15T04:09:25.739 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 -- start start 2023-11-15T04:09:25.739 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:25.739 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.738+0000 7f9d6988b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640fb820 con 0x7f9d640f5410 2023-11-15T04:09:25.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d62ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:25.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d62ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50280/0 (socket says 172.21.15.188:50280) 2023-11-15T04:09:25.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d62ffd700 1 -- 172.21.15.188:0/3709316426 learned_addr learned my addr 172.21.15.188:0/3709316426 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:25.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d62ffd700 1 -- 172.21.15.188:0/3709316426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58005040 con 0x7f9d640f5410 2023-11-15T04:09:25.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d62ffd700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9d58013510 tx=0x7f9d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:25.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d58009320 con 0x7f9d640f5410 2023-11-15T04:09:25.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d58013f00 con 0x7f9d640f5410 2023-11-15T04:09:25.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d640fb9c0 con 0x7f9d640f5410 2023-11-15T04:09:25.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d58014d90 con 0x7f9d640f5410 2023-11-15T04:09:25.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.739+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d640fbe50 con 0x7f9d640f5410 2023-11-15T04:09:25.743 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.740+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d580263a0 con 0x7f9d640f5410 2023-11-15T04:09:25.743 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.740+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d64059100 con 0x7f9d640f5410 2023-11-15T04:09:25.743 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.740+0000 7f9d4b7fe700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 0x7f9d4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:25.743 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.740+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d580413e0 con 0x7f9d640f5410 2023-11-15T04:09:25.743 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.741+0000 7f9d627fc700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 0x7f9d4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:25.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.741+0000 7f9d627fc700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 0x7f9d4c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:25.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.746+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d580126f0 con 0x7f9d640f5410 2023-11-15T04:09:25.919 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.919+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 --> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi188:172.21.15.188=smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d640f29d0 con 0x7f9d4c02ff60 2023-11-15T04:09:25.925 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.925+0000 7f9d4b7fe700 1 -- 172.21.15.188:0/3709316426 <== mgr.14164 v2:172.21.15.38:6800/3073786548 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9d640f29d0 con 0x7f9d4c02ff60 2023-11-15T04:09:25.925 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2023-11-15T04:09:25.926 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 msgr2=0x7f9d4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:25.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 0x7f9d4c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).stop 2023-11-15T04:09:25.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 msgr2=0x7f9d640fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:25.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9d58013510 tx=0x7f9d5800b040).stop 2023-11-15T04:09:25.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 shutdown_connections 2023-11-15T04:09:25.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d4c02ff60 0x7f9d4c032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:25.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 --2- 172.21.15.188:0/3709316426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d640f5410 0x7f9d640fd3a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:25.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 >> 172.21.15.188:0/3709316426 conn(0x7f9d640f0b70 msgr2=0x7f9d6405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:25.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 shutdown_connections 2023-11-15T04:09:25.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:25.927+0000 7f9d6988b700 1 -- 172.21.15.188:0/3709316426 wait complete. 2023-11-15T04:09:26.303 DEBUG:teuthology.orchestra.run.smithi188:mon.smithi188> sudo journalctl -f -n 0 -u ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188.service 2023-11-15T04:09:26.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:26.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:26.336 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:-- Logs begin at Wed 2023-11-15 03:54:39 UTC. -- 2023-11-15T04:09:27.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:25.920474+0000 mgr.smithi038.bnbogz (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi038:172.21.15.38=smithi038;smithi188:172.21.15.188=smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:09:27.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:09:25.921924+0000 mgr.smithi038.bnbogz (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi038:172.21.15.38=smithi038;smithi188:172.21.15.188=smithi188;count:2 2023-11-15T04:09:27.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:25.924899+0000 mon.smithi038 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:09:27.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:26.051225+0000 mgr.smithi038.bnbogz (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.523+0000 7ff9ed845700 1 -- 172.21.15.188:0/1542266911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 msgr2=0x7ff9e80fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.523+0000 7ff9ed845700 1 --2- 172.21.15.188:0/1542266911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e80fb3a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff9dc009fb0 tx=0x7ff9dc009b30).stop 2023-11-15T04:09:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.523+0000 7ff9ed845700 1 -- 172.21.15.188:0/1542266911 shutdown_connections 2023-11-15T04:09:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.523+0000 7ff9ed845700 1 --2- 172.21.15.188:0/1542266911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e80fb3a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 -- 172.21.15.188:0/1542266911 >> 172.21.15.188:0/1542266911 conn(0x7ff9e80f09a0 msgr2=0x7ff9e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 -- 172.21.15.188:0/1542266911 shutdown_connections 2023-11-15T04:09:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 -- 172.21.15.188:0/1542266911 wait complete. 2023-11-15T04:09:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 Processor -- start 2023-11-15T04:09:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 -- start start 2023-11-15T04:09:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.524+0000 7ff9ed845700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8108be0 con 0x7ff9e8060160 2023-11-15T04:09:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ec843700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ec843700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50308/0 (socket says 172.21.15.188:50308) 2023-11-15T04:09:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ec843700 1 -- 172.21.15.188:0/3135616873 learned_addr learned my addr 172.21.15.188:0/3135616873 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ec843700 1 -- 172.21.15.188:0/3135616873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc005040 con 0x7ff9e8060160 2023-11-15T04:09:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ec843700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff9dc013510 tx=0x7ff9dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:27.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc009320 con 0x7ff9e8060160 2023-11-15T04:09:27.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9dc013f00 con 0x7ff9e8060160 2023-11-15T04:09:27.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc014d90 con 0x7ff9e8060160 2023-11-15T04:09:27.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.525+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8108de0 con 0x7ff9e8060160 2023-11-15T04:09:27.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.526+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e8109270 con 0x7ff9e8060160 2023-11-15T04:09:27.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.526+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9dc0095f0 con 0x7ff9e8060160 2023-11-15T04:09:27.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.526+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8045ad0 con 0x7ff9e8060160 2023-11-15T04:09:27.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.526+0000 7ff9e57fa700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 0x7ff9d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:27.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.526+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9dc041000 con 0x7ff9e8060160 2023-11-15T04:09:27.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.527+0000 7ff9e7fff700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 0x7ff9d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:27.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.527+0000 7ff9e7fff700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 0x7ff9d8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff9d40096a0 tx=0x7ff9d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:27.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.532+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9dc01d7f0 con 0x7ff9e8060160 2023-11-15T04:09:27.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.735+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9e810a120 con 0x7ff9e8060160 2023-11-15T04:09:27.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.736+0000 7ff9e57fa700 1 -- 172.21.15.188:0/3135616873 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9dc010230 con 0x7ff9e8060160 2023-11-15T04:09:27.738 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:27.739 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:27.739 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.737+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 msgr2=0x7ff9d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:27.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.737+0000 7ff9ed845700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 0x7ff9d8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff9d40096a0 tx=0x7ff9d4006b40).stop 2023-11-15T04:09:27.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.737+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 msgr2=0x7ff9e81086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:27.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.737+0000 7ff9ed845700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff9dc013510 tx=0x7ff9dc00b040).stop 2023-11-15T04:09:27.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 shutdown_connections 2023-11-15T04:09:27.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d802ff60 0x7ff9d8032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:27.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 --2- 172.21.15.188:0/3135616873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8060160 0x7ff9e81086f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:27.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 >> 172.21.15.188:0/3135616873 conn(0x7ff9e80f09a0 msgr2=0x7ff9e80f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:27.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 shutdown_connections 2023-11-15T04:09:27.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:27.738+0000 7ff9ed845700 1 -- 172.21.15.188:0/3135616873 wait complete. 2023-11-15T04:09:27.741 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:28.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:27.735892+0000 mon.smithi038 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.188:0/3135616873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:29.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:29.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:29.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:28.051499+0000 mgr.smithi038.bnbogz (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:30.292 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- 172.21.15.188:0/1040836174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 msgr2=0x7fcb5c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:30.292 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 --2- 172.21.15.188:0/1040836174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c0f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fcb50009fb0 tx=0x7fcb50009b30).stop 2023-11-15T04:09:30.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- 172.21.15.188:0/1040836174 shutdown_connections 2023-11-15T04:09:30.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 --2- 172.21.15.188:0/1040836174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c0f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:30.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- 172.21.15.188:0/1040836174 >> 172.21.15.188:0/1040836174 conn(0x7fcb5c0f09a0 msgr2=0x7fcb5c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:30.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- 172.21.15.188:0/1040836174 shutdown_connections 2023-11-15T04:09:30.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- 172.21.15.188:0/1040836174 wait complete. 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 Processor -- start 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.290+0000 7fcb61e88700 1 -- start start 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb61e88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb61e88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c060280 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb60e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:30.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb60e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37002/0 (socket says 172.21.15.188:37002) 2023-11-15T04:09:30.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb60e86700 1 -- 172.21.15.188:0/2209849068 learned_addr learned my addr 172.21.15.188:0/2209849068 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:30.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.291+0000 7fcb60e86700 1 -- 172.21.15.188:0/2209849068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50005040 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb60e86700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcb50013510 tx=0x7fcb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:30.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb50009320 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c060480 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb50013f00 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb50014dc0 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.292+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c1098d0 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.293+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb50014420 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.293+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c045ad0 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.298 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.293+0000 7fcb597fa700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:30.298 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.293+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb50041050 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.298 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.293+0000 7fcb5bfff700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:30.298 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.294+0000 7fcb5bfff700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 0x7fcb4c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:30.299 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.298+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb5001cb10 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.505+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb5c0fabe0 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.505+0000 7fcb597fa700 1 -- 172.21.15.188:0/2209849068 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb5001cb10 con 0x7fcb5c0f51a0 2023-11-15T04:09:30.506 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:30.507 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:30.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 msgr2=0x7fcb4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:30.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 0x7fcb4c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).stop 2023-11-15T04:09:30.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 msgr2=0x7fcb5c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:30.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcb50013510 tx=0x7fcb5000b040).stop 2023-11-15T04:09:30.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 shutdown_connections 2023-11-15T04:09:30.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:30.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 --2- 172.21.15.188:0/2209849068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c0f51a0 0x7fcb5c1091a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:30.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 >> 172.21.15.188:0/2209849068 conn(0x7fcb5c0f09a0 msgr2=0x7fcb5c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:30.510 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 shutdown_connections 2023-11-15T04:09:30.510 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:30.507+0000 7fcb61e88700 1 -- 172.21.15.188:0/2209849068 wait complete. 2023-11-15T04:09:30.510 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:30.051839+0000 mgr.smithi038.bnbogz (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:30.506728+0000 mon.smithi038 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.188:0/2209849068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:31.904 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:31.905 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:32.052120+0000 mgr.smithi038.bnbogz (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:33.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.114+0000 7fb74f59e700 1 -- 172.21.15.188:0/1367663796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 msgr2=0x7fb7500f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:33.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.114+0000 7fb74f59e700 1 --2- 172.21.15.188:0/1367663796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7500f66c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb744009fb0 tx=0x7fb744009b30).stop 2023-11-15T04:09:33.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- 172.21.15.188:0/1367663796 shutdown_connections 2023-11-15T04:09:33.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 --2- 172.21.15.188:0/1367663796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7500f66c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:33.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- 172.21.15.188:0/1367663796 >> 172.21.15.188:0/1367663796 conn(0x7fb7500f1aa0 msgr2=0x7fb7500f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:33.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- 172.21.15.188:0/1367663796 shutdown_connections 2023-11-15T04:09:33.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- 172.21.15.188:0/1367663796 wait complete. 2023-11-15T04:09:33.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 Processor -- start 2023-11-15T04:09:33.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- start start 2023-11-15T04:09:33.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:33.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.115+0000 7fb74f59e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7501069a0 con 0x7fb7500f62a0 2023-11-15T04:09:33.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:33.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37012/0 (socket says 172.21.15.188:37012) 2023-11-15T04:09:33.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74e59c700 1 -- 172.21.15.188:0/4243187607 learned_addr learned my addr 172.21.15.188:0/4243187607 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:33.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74e59c700 1 -- 172.21.15.188:0/4243187607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744005040 con 0x7fb7500f62a0 2023-11-15T04:09:33.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74e59c700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb744013510 tx=0x7fb74400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:33.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb744009320 con 0x7fb7500f62a0 2023-11-15T04:09:33.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb744013f00 con 0x7fb7500f62a0 2023-11-15T04:09:33.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb750106ba0 con 0x7fb7500f62a0 2023-11-15T04:09:33.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb744014dc0 con 0x7fb7500f62a0 2023-11-15T04:09:33.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.116+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb750107030 con 0x7fb7500f62a0 2023-11-15T04:09:33.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.117+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb750045ad0 con 0x7fb7500f62a0 2023-11-15T04:09:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.117+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7440095f0 con 0x7fb7500f62a0 2023-11-15T04:09:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.117+0000 7fb73effd700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 0x7fb738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.117+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7440410c0 con 0x7fb7500f62a0 2023-11-15T04:09:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.117+0000 7fb74dd9b700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 0x7fb738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.118+0000 7fb74dd9b700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 0x7fb738032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb7400096a0 tx=0x7fb740006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:33.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.122+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb744014420 con 0x7fb7500f62a0 2023-11-15T04:09:33.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.329+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb750107ee0 con 0x7fb7500f62a0 2023-11-15T04:09:33.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.330+0000 7fb73effd700 1 -- 172.21.15.188:0/4243187607 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb744010230 con 0x7fb7500f62a0 2023-11-15T04:09:33.331 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:33.332 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:33.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 msgr2=0x7fb738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:33.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 0x7fb738032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb7400096a0 tx=0x7fb740006b40).stop 2023-11-15T04:09:33.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 msgr2=0x7fb7501064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:33.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb744013510 tx=0x7fb74400b040).stop 2023-11-15T04:09:33.335 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 shutdown_connections 2023-11-15T04:09:33.335 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb73802ff60 0x7fb738032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:33.335 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 --2- 172.21.15.188:0/4243187607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7500f62a0 0x7fb7501064b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:33.336 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 >> 172.21.15.188:0/4243187607 conn(0x7fb7500f1aa0 msgr2=0x7fb7500f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:33.336 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 shutdown_connections 2023-11-15T04:09:33.336 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:33.331+0000 7fb74f59e700 1 -- 172.21.15.188:0/4243187607 wait complete. 2023-11-15T04:09:33.336 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:33.331135+0000 mon.smithi038 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.188:0/4243187607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:34.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:34.765 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:34.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:34.052471+0000 mgr.smithi038.bnbogz (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:36.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 -- 172.21.15.188:0/1569910422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 msgr2=0x7ff4ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:36.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 --2- 172.21.15.188:0/1569910422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff4dc009fb0 tx=0x7ff4dc009b30).stop 2023-11-15T04:09:36.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 -- 172.21.15.188:0/1569910422 shutdown_connections 2023-11-15T04:09:36.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 --2- 172.21.15.188:0/1569910422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:36.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 -- 172.21.15.188:0/1569910422 >> 172.21.15.188:0/1569910422 conn(0x7ff4ec0f0b70 msgr2=0x7ff4ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:36.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 -- 172.21.15.188:0/1569910422 shutdown_connections 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.477+0000 7ff4f340d700 1 -- 172.21.15.188:0/1569910422 wait complete. 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f340d700 1 Processor -- start 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f340d700 1 -- start start 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f340d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f340d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec106c00 con 0x7ff4ec0f5410 2023-11-15T04:09:36.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f11a9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f11a9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37030/0 (socket says 172.21.15.188:37030) 2023-11-15T04:09:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.478+0000 7ff4f11a9700 1 -- 172.21.15.188:0/928148193 learned_addr learned my addr 172.21.15.188:0/928148193 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4f11a9700 1 -- 172.21.15.188:0/928148193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc005040 con 0x7ff4ec0f5410 2023-11-15T04:09:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4f11a9700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff4dc013510 tx=0x7ff4dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4dc009320 con 0x7ff4ec0f5410 2023-11-15T04:09:36.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec106e00 con 0x7ff4ec0f5410 2023-11-15T04:09:36.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4dc013f00 con 0x7ff4ec0f5410 2023-11-15T04:09:36.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4dc014d90 con 0x7ff4ec0f5410 2023-11-15T04:09:36.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.479+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec107290 con 0x7ff4ec0f5410 2023-11-15T04:09:36.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.480+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4dc014420 con 0x7ff4ec0f5410 2023-11-15T04:09:36.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.480+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec045ad0 con 0x7ff4ec0f5410 2023-11-15T04:09:36.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.480+0000 7ff4e1ffb700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 0x7ff4d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:36.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.480+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4dc0410a0 con 0x7ff4ec0f5410 2023-11-15T04:09:36.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.480+0000 7ff4f09a8700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 0x7ff4d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:36.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.481+0000 7ff4f09a8700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 0x7ff4d8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff4e80096a0 tx=0x7ff4e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:36.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.485+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4dc0095f0 con 0x7ff4ec0f5410 2023-11-15T04:09:36.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.691+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4ec0fae60 con 0x7ff4ec0f5410 2023-11-15T04:09:36.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.691+0000 7ff4e1ffb700 1 -- 172.21.15.188:0/928148193 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4dc01db70 con 0x7ff4ec0f5410 2023-11-15T04:09:36.692 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:36.692 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:36.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 msgr2=0x7ff4d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:36.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 0x7ff4d8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff4e80096a0 tx=0x7ff4e8006b40).stop 2023-11-15T04:09:36.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 msgr2=0x7ff4ec106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:36.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff4dc013510 tx=0x7ff4dc00b040).stop 2023-11-15T04:09:36.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 shutdown_connections 2023-11-15T04:09:36.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4d802ff60 0x7ff4d8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:36.695 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 --2- 172.21.15.188:0/928148193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ec0f5410 0x7ff4ec106710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:36.695 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 >> 172.21.15.188:0/928148193 conn(0x7ff4ec0f0b70 msgr2=0x7ff4ec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:36.695 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 shutdown_connections 2023-11-15T04:09:36.695 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:36.693+0000 7ff4f340d700 1 -- 172.21.15.188:0/928148193 wait complete. 2023-11-15T04:09:36.696 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:36.052827+0000 mgr.smithi038.bnbogz (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:36.692484+0000 mon.smithi038 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.188:0/928148193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:38.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:38.088 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:39.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 -- 172.21.15.188:0/1415905787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 msgr2=0x7fc99c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:39.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 --2- 172.21.15.188:0/1415905787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c0f55c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc990009fb0 tx=0x7fc990009b30).stop 2023-11-15T04:09:39.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 -- 172.21.15.188:0/1415905787 shutdown_connections 2023-11-15T04:09:39.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 --2- 172.21.15.188:0/1415905787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c0f55c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:39.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 -- 172.21.15.188:0/1415905787 >> 172.21.15.188:0/1415905787 conn(0x7fc99c0f09a0 msgr2=0x7fc99c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:39.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 -- 172.21.15.188:0/1415905787 shutdown_connections 2023-11-15T04:09:39.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.301+0000 7fc99bfff700 1 -- 172.21.15.188:0/1415905787 wait complete. 2023-11-15T04:09:39.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99bfff700 1 Processor -- start 2023-11-15T04:09:39.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99bfff700 1 -- start start 2023-11-15T04:09:39.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:39.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99bfff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c060220 con 0x7fc99c0f51a0 2023-11-15T04:09:39.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:39.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54874/0 (socket says 172.21.15.188:54874) 2023-11-15T04:09:39.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99affd700 1 -- 172.21.15.188:0/3925372191 learned_addr learned my addr 172.21.15.188:0/3925372191 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:39.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99affd700 1 -- 172.21.15.188:0/3925372191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc990005040 con 0x7fc99c0f51a0 2023-11-15T04:09:39.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.302+0000 7fc99affd700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc990013510 tx=0x7fc99000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc990009320 con 0x7fc99c0f51a0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c060420 con 0x7fc99c0f51a0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc990013f00 con 0x7fc99c0f51a0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc990014dc0 con 0x7fc99c0f51a0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c1098b0 con 0x7fc99c0f51a0 2023-11-15T04:09:39.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc990014420 con 0x7fc99c0f51a0 2023-11-15T04:09:39.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.303+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c059100 con 0x7fc99c0f51a0 2023-11-15T04:09:39.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.304+0000 7fc9837fe700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:39.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.304+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9900410e0 con 0x7fc99c0f51a0 2023-11-15T04:09:39.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.304+0000 7fc99a7fc700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:39.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.304+0000 7fc99a7fc700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 0x7fc984032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc98c0096a0 tx=0x7fc98c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:39.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.309+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9900095f0 con 0x7fc99c0f51a0 2023-11-15T04:09:39.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.514+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc99c0fabe0 con 0x7fc99c0f51a0 2023-11-15T04:09:39.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.514+0000 7fc9837fe700 1 -- 172.21.15.188:0/3925372191 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc99001cb10 con 0x7fc99c0f51a0 2023-11-15T04:09:39.515 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:39.515 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:39.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 msgr2=0x7fc984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:39.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 0x7fc984032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc98c0096a0 tx=0x7fc98c006b40).stop 2023-11-15T04:09:39.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 msgr2=0x7fc99c109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:39.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc990013510 tx=0x7fc99000b040).stop 2023-11-15T04:09:39.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 shutdown_connections 2023-11-15T04:09:39.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc98402ff60 0x7fc984032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:39.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 --2- 172.21.15.188:0/3925372191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc99c0f51a0 0x7fc99c109200 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:39.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 >> 172.21.15.188:0/3925372191 conn(0x7fc99c0f09a0 msgr2=0x7fc99c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:39.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 shutdown_connections 2023-11-15T04:09:39.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:39.516+0000 7fc99bfff700 1 -- 172.21.15.188:0/3925372191 wait complete. 2023-11-15T04:09:39.519 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:38.053070+0000 mgr.smithi038.bnbogz (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:40.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:39.515657+0000 mon.smithi038 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.188:0/3925372191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:40.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:40.912 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:40.053414+0000 mgr.smithi038.bnbogz (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:42.053763+0000 mgr.smithi038.bnbogz (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:44.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.222+0000 7f955877d700 1 -- 172.21.15.188:0/4023754825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 msgr2=0x7f95500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:44.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.222+0000 7f955877d700 1 --2- 172.21.15.188:0/4023754825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f95500f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9540009fb0 tx=0x7f9540009b30).stop 2023-11-15T04:09:44.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.222+0000 7f955877d700 1 -- 172.21.15.188:0/4023754825 shutdown_connections 2023-11-15T04:09:44.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.222+0000 7f955877d700 1 --2- 172.21.15.188:0/4023754825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f95500f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:44.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.222+0000 7f955877d700 1 -- 172.21.15.188:0/4023754825 >> 172.21.15.188:0/4023754825 conn(0x7f95500f0b70 msgr2=0x7f95500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:44.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 -- 172.21.15.188:0/4023754825 shutdown_connections 2023-11-15T04:09:44.505 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 -- 172.21.15.188:0/4023754825 wait complete. 2023-11-15T04:09:44.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 Processor -- start 2023-11-15T04:09:44.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 -- start start 2023-11-15T04:09:44.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:44.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f955877d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550060240 con 0x7f95500f5410 2023-11-15T04:09:44.506 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f9556519700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:44.507 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f9556519700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54902/0 (socket says 172.21.15.188:54902) 2023-11-15T04:09:44.507 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.223+0000 7f9556519700 1 -- 172.21.15.188:0/3308917342 learned_addr learned my addr 172.21.15.188:0/3308917342 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:44.507 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f9556519700 1 -- 172.21.15.188:0/3308917342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540005040 con 0x7f95500f5410 2023-11-15T04:09:44.507 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f9556519700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9540014000 tx=0x7f954000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:44.507 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95400095f0 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9550060440 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9540007e70 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9540014ea0 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.224+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9550109ae0 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.225+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9540025070 con 0x7f95500f5410 2023-11-15T04:09:44.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.225+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9550045ad0 con 0x7f95500f5410 2023-11-15T04:09:44.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.225+0000 7f9546ffd700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 0x7f953c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:44.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.225+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95400409f0 con 0x7f95500f5410 2023-11-15T04:09:44.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.225+0000 7f9555d18700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 0x7f953c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:44.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.226+0000 7f9555d18700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 0x7f953c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f954c0096a0 tx=0x7f954c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:44.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.230+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f954000f4e0 con 0x7f95500f5410 2023-11-15T04:09:44.510 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.439+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95500fae60 con 0x7f95500f5410 2023-11-15T04:09:44.510 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.440+0000 7f9546ffd700 1 -- 172.21.15.188:0/3308917342 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f954000f300 con 0x7f95500f5410 2023-11-15T04:09:44.510 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:44.510 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:44.511 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 msgr2=0x7f953c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:44.511 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 0x7f953c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f954c0096a0 tx=0x7f954c006b40).stop 2023-11-15T04:09:44.511 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 msgr2=0x7f9550109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:44.511 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9540014000 tx=0x7f954000b040).stop 2023-11-15T04:09:44.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 shutdown_connections 2023-11-15T04:09:44.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f953c02ff60 0x7f953c032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:44.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 --2- 172.21.15.188:0/3308917342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95500f5410 0x7f9550109430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:44.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.441+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 >> 172.21.15.188:0/3308917342 conn(0x7f95500f0b70 msgr2=0x7f95500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:44.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.442+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 shutdown_connections 2023-11-15T04:09:44.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:44.442+0000 7f955877d700 1 -- 172.21.15.188:0/3308917342 wait complete. 2023-11-15T04:09:44.513 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:44.054095+0000 mgr.smithi038.bnbogz (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:44.441063+0000 mon.smithi038 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.188:0/3308917342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:47.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:47.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:46.054346+0000 mgr.smithi038.bnbogz (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:48.054579+0000 mgr.smithi038.bnbogz (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:50.054822+0000 mgr.smithi038.bnbogz (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:50.207274+0000 mon.smithi038 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:09:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:50.210135+0000 mon.smithi038 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:09:52.553 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 -- 172.21.15.188:0/1040976119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 msgr2=0x7f16c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:52.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/1040976119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f90f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f16bc009fb0 tx=0x7f16bc009b30).stop 2023-11-15T04:09:52.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 -- 172.21.15.188:0/1040976119 shutdown_connections 2023-11-15T04:09:52.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/1040976119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f90f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:52.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 -- 172.21.15.188:0/1040976119 >> 172.21.15.188:0/1040976119 conn(0x7f16c80f0b70 msgr2=0x7f16c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:52.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 -- 172.21.15.188:0/1040976119 shutdown_connections 2023-11-15T04:09:52.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 -- 172.21.15.188:0/1040976119 wait complete. 2023-11-15T04:09:52.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.552+0000 7f16cdbd6700 1 Processor -- start 2023-11-15T04:09:52.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16cdbd6700 1 -- start start 2023-11-15T04:09:52.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16cdbd6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:52.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16cdbd6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80f4ff0 con 0x7f16c8060280 2023-11-15T04:09:52.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16c77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:52.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16c77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:43978/0 (socket says 172.21.15.188:43978) 2023-11-15T04:09:52.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16c77fe700 1 -- 172.21.15.188:0/2611030320 learned_addr learned my addr 172.21.15.188:0/2611030320 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:52.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.553+0000 7f16c77fe700 1 -- 172.21.15.188:0/2611030320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc005040 con 0x7f16c8060280 2023-11-15T04:09:52.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16c77fe700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f16bc013510 tx=0x7f16bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:52.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16bc009320 con 0x7f16c8060280 2023-11-15T04:09:52.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c80f5190 con 0x7f16c8060280 2023-11-15T04:09:52.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16bc013f00 con 0x7f16c8060280 2023-11-15T04:09:52.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16bc014dc0 con 0x7f16c8060280 2023-11-15T04:09:52.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.554+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c80f5620 con 0x7f16c8060280 2023-11-15T04:09:52.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16bc014420 con 0x7f16c8060280 2023-11-15T04:09:52.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c8059100 con 0x7f16c8060280 2023-11-15T04:09:52.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16affff700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:52.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16bc0410f0 con 0x7f16c8060280 2023-11-15T04:09:52.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16c6ffd700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:52.561 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.555+0000 7f16c6ffd700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 0x7f16b0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f16b80089e0 tx=0x7f16b8006ec0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:52.561 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.560+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16bc0095f0 con 0x7f16c8060280 2023-11-15T04:09:52.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.765+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16c80fbf50 con 0x7f16c8060280 2023-11-15T04:09:52.766 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.765+0000 7f16affff700 1 -- 172.21.15.188:0/2611030320 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16bc01cb10 con 0x7f16c8060280 2023-11-15T04:09:52.766 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:52.767 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:52.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 msgr2=0x7f16b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:52.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 0x7f16b0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f16b80089e0 tx=0x7f16b8006ec0).stop 2023-11-15T04:09:52.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 msgr2=0x7f16c80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:52.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f16bc013510 tx=0x7f16bc00b040).stop 2023-11-15T04:09:52.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 shutdown_connections 2023-11-15T04:09:52.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:52.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 --2- 172.21.15.188:0/2611030320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16c8060280 0x7f16c80f6b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:52.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 >> 172.21.15.188:0/2611030320 conn(0x7f16c80f0b70 msgr2=0x7f16c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:52.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 shutdown_connections 2023-11-15T04:09:52.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:52.767+0000 7f16cdbd6700 1 -- 172.21.15.188:0/2611030320 wait complete. 2023-11-15T04:09:52.772 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:52.055099+0000 mgr.smithi038.bnbogz (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:53.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:52.766616+0000 mon.smithi038 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.188:0/2611030320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:54.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:54.159 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:54.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:54.055482+0000 mgr.smithi038.bnbogz (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:55.412 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 -- 172.21.15.188:0/3948941244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 msgr2=0x7f83740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:55.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 --2- 172.21.15.188:0/3948941244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f83740f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8368009fb0 tx=0x7f8368009b30).stop 2023-11-15T04:09:55.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 -- 172.21.15.188:0/3948941244 shutdown_connections 2023-11-15T04:09:55.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 --2- 172.21.15.188:0/3948941244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f83740f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:55.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 -- 172.21.15.188:0/3948941244 >> 172.21.15.188:0/3948941244 conn(0x7f83740f0b70 msgr2=0x7f83740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:55.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 -- 172.21.15.188:0/3948941244 shutdown_connections 2023-11-15T04:09:55.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.411+0000 7f8379d6c700 1 -- 172.21.15.188:0/3948941244 wait complete. 2023-11-15T04:09:55.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.412+0000 7f8379d6c700 1 Processor -- start 2023-11-15T04:09:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.412+0000 7f8379d6c700 1 -- start start 2023-11-15T04:09:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.412+0000 7f8379d6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.412+0000 7f8379d6c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8374106c40 con 0x7f83740f5410 2023-11-15T04:09:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.412+0000 7f83737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:55.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f83737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:43998/0 (socket says 172.21.15.188:43998) 2023-11-15T04:09:55.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f83737fe700 1 -- 172.21.15.188:0/4202695832 learned_addr learned my addr 172.21.15.188:0/4202695832 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:55.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f83737fe700 1 -- 172.21.15.188:0/4202695832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368005040 con 0x7f83740f5410 2023-11-15T04:09:55.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f83737fe700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f83680089f0 tx=0x7f836800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:55.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8368009320 con 0x7f83740f5410 2023-11-15T04:09:55.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8374106e40 con 0x7f83740f5410 2023-11-15T04:09:55.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83680136c0 con 0x7f83740f5410 2023-11-15T04:09:55.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f836801ecb0 con 0x7f83740f5410 2023-11-15T04:09:55.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.413+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83741072d0 con 0x7f83740f5410 2023-11-15T04:09:55.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.414+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f836801e420 con 0x7f83740f5410 2023-11-15T04:09:55.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.414+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8374045ad0 con 0x7f83740f5410 2023-11-15T04:09:55.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.414+0000 7f835bfff700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:55.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.414+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8368040eb0 con 0x7f83740f5410 2023-11-15T04:09:55.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.414+0000 7f8372ffd700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:55.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.415+0000 7f8372ffd700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 0x7f835c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83640096a0 tx=0x7f8364006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:55.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.419+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f836801c920 con 0x7f83740f5410 2023-11-15T04:09:55.625 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.625+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8374108120 con 0x7f83740f5410 2023-11-15T04:09:55.626 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.625+0000 7f835bfff700 1 -- 172.21.15.188:0/4202695832 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f836800e2a0 con 0x7f83740f5410 2023-11-15T04:09:55.626 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:55.626 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:55.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 msgr2=0x7f835c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 0x7f835c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83640096a0 tx=0x7f8364006b40).stop 2023-11-15T04:09:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 msgr2=0x7f8374106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f83680089f0 tx=0x7f836800b040).stop 2023-11-15T04:09:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 shutdown_connections 2023-11-15T04:09:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:55.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 --2- 172.21.15.188:0/4202695832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83740f5410 0x7f8374106750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:55.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 >> 172.21.15.188:0/4202695832 conn(0x7f83740f0b70 msgr2=0x7f837405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:55.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 shutdown_connections 2023-11-15T04:09:55.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:55.627+0000 7f8379d6c700 1 -- 172.21.15.188:0/4202695832 wait complete. 2023-11-15T04:09:55.630 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:56.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:55.626586+0000 mon.smithi038 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.188:0/4202695832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:57.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:57.001 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:09:57.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:56.055836+0000 mgr.smithi038.bnbogz (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:58.198 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 -- 172.21.15.188:0/3178513304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 msgr2=0x7fd5080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:58.198 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 --2- 172.21.15.188:0/3178513304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd5080f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd504009fb0 tx=0x7fd504009b30).stop 2023-11-15T04:09:58.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 -- 172.21.15.188:0/3178513304 shutdown_connections 2023-11-15T04:09:58.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 --2- 172.21.15.188:0/3178513304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd5080f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:58.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 -- 172.21.15.188:0/3178513304 >> 172.21.15.188:0/3178513304 conn(0x7fd5080f0b70 msgr2=0x7fd5080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:58.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 -- 172.21.15.188:0/3178513304 shutdown_connections 2023-11-15T04:09:58.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 -- 172.21.15.188:0/3178513304 wait complete. 2023-11-15T04:09:58.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.197+0000 7fd50f72c700 1 Processor -- start 2023-11-15T04:09:58.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50f72c700 1 -- start start 2023-11-15T04:09:58.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50f72c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:58.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50f72c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508105b20 con 0x7fd5080f5410 2023-11-15T04:09:58.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50d4c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:58.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50d4c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:44024/0 (socket says 172.21.15.188:44024) 2023-11-15T04:09:58.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50d4c8700 1 -- 172.21.15.188:0/269030352 learned_addr learned my addr 172.21.15.188:0/269030352 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:09:58.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50d4c8700 1 -- 172.21.15.188:0/269030352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd504005040 con 0x7fd5080f5410 2023-11-15T04:09:58.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.198+0000 7fd50d4c8700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd504013510 tx=0x7fd50400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:58.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.199+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd504009320 con 0x7fd5080f5410 2023-11-15T04:09:58.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.199+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd508105d20 con 0x7fd5080f5410 2023-11-15T04:09:58.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.199+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5081061b0 con 0x7fd5080f5410 2023-11-15T04:09:58.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.199+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd504013f00 con 0x7fd5080f5410 2023-11-15T04:09:58.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.199+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd504014ce0 con 0x7fd5080f5410 2023-11-15T04:09:58.206 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.200+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e8004fa0 con 0x7fd5080f5410 2023-11-15T04:09:58.206 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.200+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd504014420 con 0x7fd5080f5410 2023-11-15T04:09:58.206 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.200+0000 7fd4fdffb700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 0x7fd4f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:09:58.206 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.200+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd504040f50 con 0x7fd5080f5410 2023-11-15T04:09:58.206 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.200+0000 7fd50ccc7700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 0x7fd4f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:09:58.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.201+0000 7fd50ccc7700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 0x7fd4f4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd4f80096a0 tx=0x7fd4f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:09:58.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.205+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd504014e40 con 0x7fd5080f5410 2023-11-15T04:09:58.411 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.410+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4e8005670 con 0x7fd5080f5410 2023-11-15T04:09:58.412 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.411+0000 7fd4fdffb700 1 -- 172.21.15.188:0/269030352 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd504012750 con 0x7fd5080f5410 2023-11-15T04:09:58.412 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:09:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:09:58.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.412+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 msgr2=0x7fd4f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:58.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.412+0000 7fd50f72c700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 0x7fd4f4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd4f80096a0 tx=0x7fd4f8006b40).stop 2023-11-15T04:09:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 msgr2=0x7fd508105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:09:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd504013510 tx=0x7fd50400b040).stop 2023-11-15T04:09:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 shutdown_connections 2023-11-15T04:09:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd4f402ff60 0x7fd4f4032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:58.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 --2- 172.21.15.188:0/269030352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5080f5410 0x7fd508105630 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:09:58.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 >> 172.21.15.188:0/269030352 conn(0x7fd5080f0b70 msgr2=0x7fd5080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:09:58.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 shutdown_connections 2023-11-15T04:09:58.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:09:58.413+0000 7fd50f72c700 1 -- 172.21.15.188:0/269030352 wait complete. 2023-11-15T04:09:58.417 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:09:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:09:58.056156+0000 mgr.smithi038.bnbogz (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:09:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:09:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:09:58.412497+0000 mon.smithi038 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.188:0/269030352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:09:59.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:09:59.828 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:01.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 -- 172.21.15.188:0/2842396124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 msgr2=0x7fe0d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 --2- 172.21.15.188:0/2842396124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d00f55c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe0c0009fb0 tx=0x7fe0c0009b30).stop 2023-11-15T04:10:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 -- 172.21.15.188:0/2842396124 shutdown_connections 2023-11-15T04:10:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 --2- 172.21.15.188:0/2842396124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d00f55c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 -- 172.21.15.188:0/2842396124 >> 172.21.15.188:0/2842396124 conn(0x7fe0d00f09a0 msgr2=0x7fe0d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 -- 172.21.15.188:0/2842396124 shutdown_connections 2023-11-15T04:10:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.026+0000 7fe0d5089700 1 -- 172.21.15.188:0/2842396124 wait complete. 2023-11-15T04:10:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0d5089700 1 Processor -- start 2023-11-15T04:10:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0d5089700 1 -- start start 2023-11-15T04:10:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0d5089700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0d5089700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0060240 con 0x7fe0d00f51a0 2023-11-15T04:10:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39420/0 (socket says 172.21.15.188:39420) 2023-11-15T04:10:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0cffff700 1 -- 172.21.15.188:0/4276173512 learned_addr learned my addr 172.21.15.188:0/4276173512 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.027+0000 7fe0cffff700 1 -- 172.21.15.188:0/4276173512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0005040 con 0x7fe0d00f51a0 2023-11-15T04:10:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0cffff700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe0c0013510 tx=0x7fe0c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c0009320 con 0x7fe0d00f51a0 2023-11-15T04:10:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d0060440 con 0x7fe0d00f51a0 2023-11-15T04:10:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0c0013f00 con 0x7fe0d00f51a0 2023-11-15T04:10:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c0014dc0 con 0x7fe0d00f51a0 2023-11-15T04:10:01.033 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.028+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d0109860 con 0x7fe0d00f51a0 2023-11-15T04:10:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.029+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0c001d3d0 con 0x7fe0d00f51a0 2023-11-15T04:10:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.029+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0045ad0 con 0x7fe0d00f51a0 2023-11-15T04:10:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.029+0000 7fe0ccff9700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.029+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0c0041340 con 0x7fe0d00f51a0 2023-11-15T04:10:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.029+0000 7fe0cf7fe700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:01.035 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.030+0000 7fe0cf7fe700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 0x7fe0b8032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe0c40096a0 tx=0x7fe0c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:01.035 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.035+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0c001ccf0 con 0x7fe0d00f51a0 2023-11-15T04:10:01.243 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.242+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0d010a080 con 0x7fe0d00f51a0 2023-11-15T04:10:01.243 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.243+0000 7fe0ccff9700 1 -- 172.21.15.188:0/4276173512 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0c002b030 con 0x7fe0d00f51a0 2023-11-15T04:10:01.244 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:01.244 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:01.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 msgr2=0x7fe0b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:01.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 0x7fe0b8032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe0c40096a0 tx=0x7fe0c4006b40).stop 2023-11-15T04:10:01.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 msgr2=0x7fe0d01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe0c0013510 tx=0x7fe0c000b040).stop 2023-11-15T04:10:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 shutdown_connections 2023-11-15T04:10:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe0b802ff60 0x7fe0b8032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 --2- 172.21.15.188:0/4276173512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d00f51a0 0x7fe0d01091b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.244+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 >> 172.21.15.188:0/4276173512 conn(0x7fe0d00f09a0 msgr2=0x7fe0d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.245+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 shutdown_connections 2023-11-15T04:10:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:01.245+0000 7fe0d5089700 1 -- 172.21.15.188:0/4276173512 wait complete. 2023-11-15T04:10:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:00.056502+0000 mgr.smithi038.bnbogz (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:02.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:01.244136+0000 mon.smithi038 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.188:0/4276173512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:02.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:02.665 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:03.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:02.056859+0000 mgr.smithi038.bnbogz (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:03.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 -- 172.21.15.188:0/2168088324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 msgr2=0x7f5fb00f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 --2- 172.21.15.188:0/2168088324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb00f90d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5f98009fb0 tx=0x7f5f98009b30).stop 2023-11-15T04:10:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 -- 172.21.15.188:0/2168088324 shutdown_connections 2023-11-15T04:10:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 --2- 172.21.15.188:0/2168088324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb00f90d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 -- 172.21.15.188:0/2168088324 >> 172.21.15.188:0/2168088324 conn(0x7f5fb00f0b70 msgr2=0x7f5fb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 -- 172.21.15.188:0/2168088324 shutdown_connections 2023-11-15T04:10:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.811+0000 7f5fb675d700 1 -- 172.21.15.188:0/2168088324 wait complete. 2023-11-15T04:10:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5fb675d700 1 Processor -- start 2023-11-15T04:10:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5fb675d700 1 -- start start 2023-11-15T04:10:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5fb675d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5fb675d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0106bc0 con 0x7f5fb0060280 2023-11-15T04:10:03.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5faffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:03.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5faffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39432/0 (socket says 172.21.15.188:39432) 2023-11-15T04:10:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.812+0000 7f5faffff700 1 -- 172.21.15.188:0/3254270283 learned_addr learned my addr 172.21.15.188:0/3254270283 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5faffff700 1 -- 172.21.15.188:0/3254270283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98005040 con 0x7f5fb0060280 2023-11-15T04:10:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5faffff700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5f98009830 tx=0x7f5f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f98009320 con 0x7f5fb0060280 2023-11-15T04:10:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb0106dc0 con 0x7f5fb0060280 2023-11-15T04:10:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f980136c0 con 0x7f5fb0060280 2023-11-15T04:10:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9801ee90 con 0x7f5fb0060280 2023-11-15T04:10:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.813+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb0107250 con 0x7f5fb0060280 2023-11-15T04:10:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.814+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f98014440 con 0x7f5fb0060280 2023-11-15T04:10:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.814+0000 7f5facff9700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 0x7f5f9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.814+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f98040ef0 con 0x7f5fb0060280 2023-11-15T04:10:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.814+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0045ad0 con 0x7f5fb0060280 2023-11-15T04:10:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.814+0000 7f5faf7fe700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 0x7f5f9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.815+0000 7f5faf7fe700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 0x7f5f9c0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5fa00096a0 tx=0x7f5fa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:03.820 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:03.819+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f9801c910 con 0x7f5fb0060280 2023-11-15T04:10:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.027+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fb00fbf30 con 0x7f5fb0060280 2023-11-15T04:10:04.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.027+0000 7f5facff9700 1 -- 172.21.15.188:0/3254270283 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f9801c320 con 0x7f5fb0060280 2023-11-15T04:10:04.028 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:04.029 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:04.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 msgr2=0x7f5f9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 0x7f5f9c0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5fa00096a0 tx=0x7f5fa0006b40).stop 2023-11-15T04:10:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 msgr2=0x7f5fb01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5f98009830 tx=0x7f5f9800b040).stop 2023-11-15T04:10:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 shutdown_connections 2023-11-15T04:10:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5f9c02ff10 0x7f5f9c0323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 --2- 172.21.15.188:0/3254270283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb0060280 0x7f5fb01066d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 >> 172.21.15.188:0/3254270283 conn(0x7f5fb00f0b70 msgr2=0x7f5fb00f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 shutdown_connections 2023-11-15T04:10:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:04.029+0000 7f5fb675d700 1 -- 172.21.15.188:0/3254270283 wait complete. 2023-11-15T04:10:04.032 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:04.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:04.028745+0000 mon.smithi038 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.188:0/3254270283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:05.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:05.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:04.057156+0000 mgr.smithi038.bnbogz (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:06.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 -- 172.21.15.188:0/2565010345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 msgr2=0x7f9c900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:06.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 --2- 172.21.15.188:0/2565010345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c900f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9c80009fb0 tx=0x7f9c80009b30).stop 2023-11-15T04:10:06.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 -- 172.21.15.188:0/2565010345 shutdown_connections 2023-11-15T04:10:06.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 --2- 172.21.15.188:0/2565010345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c900f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:06.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 -- 172.21.15.188:0/2565010345 >> 172.21.15.188:0/2565010345 conn(0x7f9c900f1c70 msgr2=0x7f9c900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:06.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 -- 172.21.15.188:0/2565010345 shutdown_connections 2023-11-15T04:10:06.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.578+0000 7f9c961ec700 1 -- 172.21.15.188:0/2565010345 wait complete. 2023-11-15T04:10:06.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.579+0000 7f9c961ec700 1 Processor -- start 2023-11-15T04:10:06.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.579+0000 7f9c961ec700 1 -- start start 2023-11-15T04:10:06.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.579+0000 7f9c961ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:06.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.579+0000 7f9c961ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90106c20 con 0x7f9c900f6510 2023-11-15T04:10:06.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c8f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:06.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c8f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39460/0 (socket says 172.21.15.188:39460) 2023-11-15T04:10:06.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c8f7fe700 1 -- 172.21.15.188:0/3412016135 learned_addr learned my addr 172.21.15.188:0/3412016135 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:06.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c8f7fe700 1 -- 172.21.15.188:0/3412016135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c80005040 con 0x7f9c900f6510 2023-11-15T04:10:06.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c8f7fe700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9c80013510 tx=0x7f9c8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:06.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c80009320 con 0x7f9c900f6510 2023-11-15T04:10:06.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c90106e20 con 0x7f9c900f6510 2023-11-15T04:10:06.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c80013f00 con 0x7f9c900f6510 2023-11-15T04:10:06.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c80014d90 con 0x7f9c900f6510 2023-11-15T04:10:06.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.580+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c901072b0 con 0x7f9c900f6510 2023-11-15T04:10:06.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.581+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c800095f0 con 0x7f9c900f6510 2023-11-15T04:10:06.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.581+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c90045ad0 con 0x7f9c900f6510 2023-11-15T04:10:06.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.581+0000 7f9c77fff700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:06.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.581+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c800410c0 con 0x7f9c900f6510 2023-11-15T04:10:06.587 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.581+0000 7f9c8effd700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:06.587 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.582+0000 7f9c8effd700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 0x7f9c78032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9c840096a0 tx=0x7f9c84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:06.587 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.586+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c80014420 con 0x7f9c900f6510 2023-11-15T04:10:06.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.793+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c900fbf60 con 0x7f9c900f6510 2023-11-15T04:10:06.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.794+0000 7f9c77fff700 1 -- 172.21.15.188:0/3412016135 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c8001d7f0 con 0x7f9c900f6510 2023-11-15T04:10:06.795 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:06.795 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:06.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 msgr2=0x7f9c78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:06.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 0x7f9c78032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9c840096a0 tx=0x7f9c84006b40).stop 2023-11-15T04:10:06.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 msgr2=0x7f9c90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:06.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9c80013510 tx=0x7f9c8000b040).stop 2023-11-15T04:10:06.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 shutdown_connections 2023-11-15T04:10:06.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c7802ff60 0x7f9c78032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:06.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 --2- 172.21.15.188:0/3412016135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c900f6510 0x7f9c90106730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:06.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 >> 172.21.15.188:0/3412016135 conn(0x7f9c900f1c70 msgr2=0x7f9c900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:06.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 shutdown_connections 2023-11-15T04:10:06.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:06.795+0000 7f9c961ec700 1 -- 172.21.15.188:0/3412016135 wait complete. 2023-11-15T04:10:06.799 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:06.057460+0000 mgr.smithi038.bnbogz (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:06.795135+0000 mon.smithi038 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.188:0/3412016135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:08.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:08.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:09.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 -- 172.21.15.188:0/501149976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 msgr2=0x7f4c4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:09.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 --2- 172.21.15.188:0/501149976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c0f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4c40009fb0 tx=0x7f4c40009b30).stop 2023-11-15T04:10:09.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 -- 172.21.15.188:0/501149976 shutdown_connections 2023-11-15T04:10:09.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 --2- 172.21.15.188:0/501149976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c0f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:09.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 -- 172.21.15.188:0/501149976 >> 172.21.15.188:0/501149976 conn(0x7f4c4c0f0b70 msgr2=0x7f4c4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:09.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 -- 172.21.15.188:0/501149976 shutdown_connections 2023-11-15T04:10:09.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.363+0000 7f4c5324e700 1 -- 172.21.15.188:0/501149976 wait complete. 2023-11-15T04:10:09.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c5324e700 1 Processor -- start 2023-11-15T04:10:09.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c5324e700 1 -- start start 2023-11-15T04:10:09.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c5324e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:09.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c5324e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c060220 con 0x7f4c4c0f5410 2023-11-15T04:10:09.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c50fea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:09.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c50fea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56296/0 (socket says 172.21.15.188:56296) 2023-11-15T04:10:09.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.364+0000 7f4c50fea700 1 -- 172.21.15.188:0/559728701 learned_addr learned my addr 172.21.15.188:0/559728701 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:09.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c50fea700 1 -- 172.21.15.188:0/559728701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40005040 con 0x7f4c4c0f5410 2023-11-15T04:10:09.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c50fea700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f4c40013510 tx=0x7f4c4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:09.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c40009320 con 0x7f4c4c0f5410 2023-11-15T04:10:09.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c060420 con 0x7f4c4c0f5410 2023-11-15T04:10:09.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c40013f00 con 0x7f4c4c0f5410 2023-11-15T04:10:09.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c40014d90 con 0x7f4c4c0f5410 2023-11-15T04:10:09.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.365+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c109b00 con 0x7f4c4c0f5410 2023-11-15T04:10:09.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.366+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c400263a0 con 0x7f4c4c0f5410 2023-11-15T04:10:09.371 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.366+0000 7f4c497fa700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 0x7f4c3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:09.371 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.366+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c40041090 con 0x7f4c4c0f5410 2023-11-15T04:10:09.371 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.366+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c059100 con 0x7f4c4c0f5410 2023-11-15T04:10:09.371 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.366+0000 7f4c4bfff700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 0x7f4c3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:09.371 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.367+0000 7f4c4bfff700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 0x7f4c3c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f4c380096a0 tx=0x7f4c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:09.372 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.372+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c40012920 con 0x7f4c4c0f5410 2023-11-15T04:10:09.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.578+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c4c045ad0 con 0x7f4c4c0f5410 2023-11-15T04:10:09.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.578+0000 7f4c497fa700 1 -- 172.21.15.188:0/559728701 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c4002b020 con 0x7f4c4c0f5410 2023-11-15T04:10:09.580 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:09.580 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:09.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 msgr2=0x7f4c3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:09.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 0x7f4c3c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f4c380096a0 tx=0x7f4c38006b40).stop 2023-11-15T04:10:09.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 msgr2=0x7f4c4c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:09.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f4c40013510 tx=0x7f4c4000b040).stop 2023-11-15T04:10:09.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 shutdown_connections 2023-11-15T04:10:09.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c3c02ff60 0x7f4c3c032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:09.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 --2- 172.21.15.188:0/559728701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c109450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:09.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 >> 172.21.15.188:0/559728701 conn(0x7f4c4c0f0b70 msgr2=0x7f4c4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:09.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 shutdown_connections 2023-11-15T04:10:09.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:09.580+0000 7f4c5324e700 1 -- 172.21.15.188:0/559728701 wait complete. 2023-11-15T04:10:09.583 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:08.057718+0000 mgr.smithi038.bnbogz (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:10.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:10 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:09.579806+0000 mon.smithi038 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.188:0/559728701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:10.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:10.961 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:10.058086+0000 mgr.smithi038.bnbogz (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:12.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.161+0000 7f4806a7b700 1 -- 172.21.15.188:0/3689080854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 msgr2=0x7f48000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:12.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.161+0000 7f4806a7b700 1 --2- 172.21.15.188:0/3689080854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48000f55c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f47f0009fb0 tx=0x7f47f0009b30).stop 2023-11-15T04:10:12.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.161+0000 7f4806a7b700 1 -- 172.21.15.188:0/3689080854 shutdown_connections 2023-11-15T04:10:12.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.161+0000 7f4806a7b700 1 --2- 172.21.15.188:0/3689080854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48000f55c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:12.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.161+0000 7f4806a7b700 1 -- 172.21.15.188:0/3689080854 >> 172.21.15.188:0/3689080854 conn(0x7f48000f09a0 msgr2=0x7f48000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:12.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 -- 172.21.15.188:0/3689080854 shutdown_connections 2023-11-15T04:10:12.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 -- 172.21.15.188:0/3689080854 wait complete. 2023-11-15T04:10:12.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 Processor -- start 2023-11-15T04:10:12.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 -- start start 2023-11-15T04:10:12.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:12.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.162+0000 7f4806a7b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4800060220 con 0x7f48000f51a0 2023-11-15T04:10:12.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4805a79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:12.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4805a79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56310/0 (socket says 172.21.15.188:56310) 2023-11-15T04:10:12.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4805a79700 1 -- 172.21.15.188:0/4248881843 learned_addr learned my addr 172.21.15.188:0/4248881843 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:12.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4805a79700 1 -- 172.21.15.188:0/4248881843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0005040 con 0x7f48000f51a0 2023-11-15T04:10:12.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4805a79700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f47f0013510 tx=0x7f47f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:12.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0009320 con 0x7f48000f51a0 2023-11-15T04:10:12.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4800060420 con 0x7f48000f51a0 2023-11-15T04:10:12.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f0013f00 con 0x7f48000f51a0 2023-11-15T04:10:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0014dc0 con 0x7f48000f51a0 2023-11-15T04:10:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.163+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4800109880 con 0x7f48000f51a0 2023-11-15T04:10:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.164+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47f0014420 con 0x7f48000f51a0 2023-11-15T04:10:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.164+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4800059100 con 0x7f48000f51a0 2023-11-15T04:10:12.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.165+0000 7f47f67fc700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 0x7f47ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:12.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.165+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47f00410a0 con 0x7f48000f51a0 2023-11-15T04:10:12.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.165+0000 7f4805278700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 0x7f47ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:12.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.165+0000 7f4805278700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 0x7f47ec032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f47fc0096a0 tx=0x7f47fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:12.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.169+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47f00095f0 con 0x7f48000f51a0 2023-11-15T04:10:12.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.374+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48000fabe0 con 0x7f48000f51a0 2023-11-15T04:10:12.374 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.374+0000 7f47f67fc700 1 -- 172.21.15.188:0/4248881843 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47f001cb10 con 0x7f48000f51a0 2023-11-15T04:10:12.374 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:12.375 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:12.375 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 msgr2=0x7f47ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:12.376 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 0x7f47ec032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f47fc0096a0 tx=0x7f47fc006b40).stop 2023-11-15T04:10:12.376 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 msgr2=0x7f48001091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:12.376 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f47f0013510 tx=0x7f47f000b040).stop 2023-11-15T04:10:12.376 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 shutdown_connections 2023-11-15T04:10:12.377 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47ec02ff60 0x7f47ec032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:12.377 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 --2- 172.21.15.188:0/4248881843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48000f51a0 0x7f48001091d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:12.377 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 >> 172.21.15.188:0/4248881843 conn(0x7f48000f09a0 msgr2=0x7f48000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:12.377 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 shutdown_connections 2023-11-15T04:10:12.377 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:12.376+0000 7f4806a7b700 1 -- 172.21.15.188:0/4248881843 wait complete. 2023-11-15T04:10:12.378 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:12.058447+0000 mgr.smithi038.bnbogz (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:12.374944+0000 mon.smithi038 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.188:0/4248881843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:13.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:13.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:14.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 -- 172.21.15.188:0/1310609497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 msgr2=0x7ff7cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:14.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 --2- 172.21.15.188:0/1310609497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc0f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff7c0009fb0 tx=0x7ff7c0009b30).stop 2023-11-15T04:10:14.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 -- 172.21.15.188:0/1310609497 shutdown_connections 2023-11-15T04:10:14.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 --2- 172.21.15.188:0/1310609497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc0f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:14.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 -- 172.21.15.188:0/1310609497 >> 172.21.15.188:0/1310609497 conn(0x7ff7cc0f1c70 msgr2=0x7ff7cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:14.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 -- 172.21.15.188:0/1310609497 shutdown_connections 2023-11-15T04:10:14.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.938+0000 7ff7d4007700 1 -- 172.21.15.188:0/1310609497 wait complete. 2023-11-15T04:10:14.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d4007700 1 Processor -- start 2023-11-15T04:10:14.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d4007700 1 -- start start 2023-11-15T04:10:14.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d4007700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:14.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d4007700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc106c20 con 0x7ff7cc0f6510 2023-11-15T04:10:14.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d1da3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:14.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d1da3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56336/0 (socket says 172.21.15.188:56336) 2023-11-15T04:10:14.943 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d1da3700 1 -- 172.21.15.188:0/922744200 learned_addr learned my addr 172.21.15.188:0/922744200 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:14.943 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.939+0000 7ff7d1da3700 1 -- 172.21.15.188:0/922744200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0005040 con 0x7ff7cc0f6510 2023-11-15T04:10:14.943 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7d1da3700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff7c0013510 tx=0x7ff7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:14.943 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0009320 con 0x7ff7cc0f6510 2023-11-15T04:10:14.943 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc106e20 con 0x7ff7cc0f6510 2023-11-15T04:10:14.944 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc1072b0 con 0x7ff7cc0f6510 2023-11-15T04:10:14.944 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c0013f00 con 0x7ff7cc0f6510 2023-11-15T04:10:14.944 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.940+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0014d60 con 0x7ff7cc0f6510 2023-11-15T04:10:14.945 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.941+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc045ad0 con 0x7ff7cc0f6510 2023-11-15T04:10:14.945 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.941+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7c00095f0 con 0x7ff7cc0f6510 2023-11-15T04:10:14.946 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.941+0000 7ff7be7fc700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 0x7ff7b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:14.946 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.941+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7c0040fd0 con 0x7ff7cc0f6510 2023-11-15T04:10:14.946 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.941+0000 7ff7d15a2700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 0x7ff7b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:14.946 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.942+0000 7ff7d15a2700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 0x7ff7b8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff7c80096a0 tx=0x7ff7c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:14.946 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:14.945+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7c0014420 con 0x7ff7cc0f6510 2023-11-15T04:10:15.151 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.151+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7cc0fbf60 con 0x7ff7cc0f6510 2023-11-15T04:10:15.152 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.152+0000 7ff7be7fc700 1 -- 172.21.15.188:0/922744200 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7c0012980 con 0x7ff7cc0f6510 2023-11-15T04:10:15.152 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:15.152 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:15.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 msgr2=0x7ff7b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:15.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 0x7ff7b8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff7c80096a0 tx=0x7ff7c8006b40).stop 2023-11-15T04:10:15.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 msgr2=0x7ff7cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:15.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff7c0013510 tx=0x7ff7c000b040).stop 2023-11-15T04:10:15.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 shutdown_connections 2023-11-15T04:10:15.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff7b802ff60 0x7ff7b8032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:15.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 --2- 172.21.15.188:0/922744200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7cc0f6510 0x7ff7cc106730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:15.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 >> 172.21.15.188:0/922744200 conn(0x7ff7cc0f1c70 msgr2=0x7ff7cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:15.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 shutdown_connections 2023-11-15T04:10:15.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:15.154+0000 7ff7d4007700 1 -- 172.21.15.188:0/922744200 wait complete. 2023-11-15T04:10:15.156 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:14.058707+0000 mgr.smithi038.bnbogz (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:16.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:16.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:16.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:15.152423+0000 mon.smithi038 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.188:0/922744200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:16.059014+0000 mgr.smithi038.bnbogz (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:17.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.734+0000 7fefc6e15700 1 -- 172.21.15.188:0/1697958725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 msgr2=0x7fefc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:17.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.734+0000 7fefc6e15700 1 --2- 172.21.15.188:0/1697958725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc00f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fefb4009fb0 tx=0x7fefb4009b30).stop 2023-11-15T04:10:17.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- 172.21.15.188:0/1697958725 shutdown_connections 2023-11-15T04:10:17.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 --2- 172.21.15.188:0/1697958725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc00f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:17.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- 172.21.15.188:0/1697958725 >> 172.21.15.188:0/1697958725 conn(0x7fefc00f1c70 msgr2=0x7fefc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:17.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- 172.21.15.188:0/1697958725 shutdown_connections 2023-11-15T04:10:17.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- 172.21.15.188:0/1697958725 wait complete. 2023-11-15T04:10:17.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 Processor -- start 2023-11-15T04:10:17.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- start start 2023-11-15T04:10:17.737 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:17.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.735+0000 7fefc6e15700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc0069e20 con 0x7fefc00f6510 2023-11-15T04:10:17.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc4bb1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:17.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc4bb1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56360/0 (socket says 172.21.15.188:56360) 2023-11-15T04:10:17.738 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc4bb1700 1 -- 172.21.15.188:0/2455460701 learned_addr learned my addr 172.21.15.188:0/2455460701 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:17.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc4bb1700 1 -- 172.21.15.188:0/2455460701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4005040 con 0x7fefc00f6510 2023-11-15T04:10:17.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc4bb1700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fefb4013510 tx=0x7fefb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:17.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb4009320 con 0x7fefc00f6510 2023-11-15T04:10:17.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc0069fc0 con 0x7fefc00f6510 2023-11-15T04:10:17.740 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefb4013f00 con 0x7fefc00f6510 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb4014d90 con 0x7fefc00f6510 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.736+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc006a450 con 0x7fefc00f6510 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.737+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefb40095f0 con 0x7fefc00f6510 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.737+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc0059100 con 0x7fefc00f6510 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.737+0000 7fefbd7fa700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 0x7fefb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:17.741 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.737+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefb40410a0 con 0x7fefc00f6510 2023-11-15T04:10:17.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.738+0000 7fefbffff700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 0x7fefb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:17.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.738+0000 7fefbffff700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 0x7fefb0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fefac0096a0 tx=0x7fefac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:17.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.742+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefb4014420 con 0x7fefc00f6510 2023-11-15T04:10:17.948 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.949+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefc00fbf60 con 0x7fefc00f6510 2023-11-15T04:10:17.949 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.949+0000 7fefbd7fa700 1 -- 172.21.15.188:0/2455460701 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefb401d7f0 con 0x7fefc00f6510 2023-11-15T04:10:17.949 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:17.950 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:17.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.950+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 msgr2=0x7fefb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:17.951 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.950+0000 7fefc6e15700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 0x7fefb0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fefac0096a0 tx=0x7fefac006b40).stop 2023-11-15T04:10:17.951 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.950+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 msgr2=0x7fefc006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:17.951 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.950+0000 7fefc6e15700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fefb4013510 tx=0x7fefb400b040).stop 2023-11-15T04:10:17.951 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 shutdown_connections 2023-11-15T04:10:17.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fefb002ff60 0x7fefb0032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:17.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 --2- 172.21.15.188:0/2455460701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefc00f6510 0x7fefc006b9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:17.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 >> 172.21.15.188:0/2455460701 conn(0x7fefc00f1c70 msgr2=0x7fefc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:17.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 shutdown_connections 2023-11-15T04:10:17.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:17.951+0000 7fefc6e15700 1 -- 172.21.15.188:0/2455460701 wait complete. 2023-11-15T04:10:17.954 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:18.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:18 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:17.949527+0000 mon.smithi038 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.188:0/2455460701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:19.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:19.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:18.059329+0000 mgr.smithi038.bnbogz (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:20.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 -- 172.21.15.188:0/2163007972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 msgr2=0x7fa294069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:20.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/2163007972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa294069e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa288009fb0 tx=0x7fa288009b30).stop 2023-11-15T04:10:20.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 -- 172.21.15.188:0/2163007972 shutdown_connections 2023-11-15T04:10:20.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/2163007972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa294069e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:20.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 -- 172.21.15.188:0/2163007972 >> 172.21.15.188:0/2163007972 conn(0x7fa2940f1c70 msgr2=0x7fa2940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:20.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 -- 172.21.15.188:0/2163007972 shutdown_connections 2023-11-15T04:10:20.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.563+0000 7fa29ad8e700 1 -- 172.21.15.188:0/2163007972 wait complete. 2023-11-15T04:10:20.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.564+0000 7fa29ad8e700 1 Processor -- start 2023-11-15T04:10:20.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.564+0000 7fa29ad8e700 1 -- start start 2023-11-15T04:10:20.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.564+0000 7fa29ad8e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:20.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.564+0000 7fa29ad8e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294108dd0 con 0x7fa29406b960 2023-11-15T04:10:20.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa298b2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:20.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa298b2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58242/0 (socket says 172.21.15.188:58242) 2023-11-15T04:10:20.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa298b2a700 1 -- 172.21.15.188:0/701517297 learned_addr learned my addr 172.21.15.188:0/701517297 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:20.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa298b2a700 1 -- 172.21.15.188:0/701517297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288005040 con 0x7fa29406b960 2023-11-15T04:10:20.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa298b2a700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa288013510 tx=0x7fa28800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:20.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa288009320 con 0x7fa29406b960 2023-11-15T04:10:20.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa294108fd0 con 0x7fa29406b960 2023-11-15T04:10:20.569 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa288013f00 con 0x7fa29406b960 2023-11-15T04:10:20.569 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa288014dc0 con 0x7fa29406b960 2023-11-15T04:10:20.569 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.565+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa294109460 con 0x7fa29406b960 2023-11-15T04:10:20.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.566+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2880095f0 con 0x7fa29406b960 2023-11-15T04:10:20.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.566+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294045ad0 con 0x7fa29406b960 2023-11-15T04:10:20.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.566+0000 7fa2917fa700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 0x7fa284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:20.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.566+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa288041040 con 0x7fa29406b960 2023-11-15T04:10:20.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.567+0000 7fa293fff700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 0x7fa284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:20.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.567+0000 7fa293fff700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 0x7fa284032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fa2800096a0 tx=0x7fa280006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:20.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.571+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa288014420 con 0x7fa29406b960 2023-11-15T04:10:20.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.775+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa29410a310 con 0x7fa29406b960 2023-11-15T04:10:20.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.775+0000 7fa2917fa700 1 -- 172.21.15.188:0/701517297 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa288010230 con 0x7fa29406b960 2023-11-15T04:10:20.776 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:20.776 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:20.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 msgr2=0x7fa284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:20.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 0x7fa284032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fa2800096a0 tx=0x7fa280006b40).stop 2023-11-15T04:10:20.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 msgr2=0x7fa2941088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:20.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa288013510 tx=0x7fa28800b040).stop 2023-11-15T04:10:20.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 shutdown_connections 2023-11-15T04:10:20.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa28402ff60 0x7fa284032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:20.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 --2- 172.21.15.188:0/701517297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa29406b960 0x7fa2941088e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:20.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 >> 172.21.15.188:0/701517297 conn(0x7fa2940f1c70 msgr2=0x7fa2940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:20.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 shutdown_connections 2023-11-15T04:10:20.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:20.777+0000 7fa29ad8e700 1 -- 172.21.15.188:0/701517297 wait complete. 2023-11-15T04:10:20.779 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:20.059681+0000 mgr.smithi038.bnbogz (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:20.775941+0000 mon.smithi038 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.188:0/701517297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:22.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:22.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:22.059908+0000 mgr.smithi038.bnbogz (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:23.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- 172.21.15.188:0/4143309875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 msgr2=0x7f5ff40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:23.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/4143309875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff40f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5ff0009fb0 tx=0x7f5ff0009b30).stop 2023-11-15T04:10:23.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- 172.21.15.188:0/4143309875 shutdown_connections 2023-11-15T04:10:23.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/4143309875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff40f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- 172.21.15.188:0/4143309875 >> 172.21.15.188:0/4143309875 conn(0x7f5ff40f09a0 msgr2=0x7f5ff40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- 172.21.15.188:0/4143309875 shutdown_connections 2023-11-15T04:10:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- 172.21.15.188:0/4143309875 wait complete. 2023-11-15T04:10:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 Processor -- start 2023-11-15T04:10:23.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- start start 2023-11-15T04:10:23.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:23.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.581+0000 7f5ffa87b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff4106fe0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ff9879700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:23.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ff9879700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58246/0 (socket says 172.21.15.188:58246) 2023-11-15T04:10:23.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ff9879700 1 -- 172.21.15.188:0/2069213650 learned_addr learned my addr 172.21.15.188:0/2069213650 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:23.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ff9879700 1 -- 172.21.15.188:0/2069213650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff0005040 con 0x7f5ff40f51a0 2023-11-15T04:10:23.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ff9879700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5ff0009830 tx=0x7f5ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:23.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff0009320 con 0x7f5ff40f51a0 2023-11-15T04:10:23.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff4060220 con 0x7f5ff40f51a0 2023-11-15T04:10:23.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ff00136c0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff001ee90 con 0x7f5ff40f51a0 2023-11-15T04:10:23.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.582+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff410bae0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.583+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ff001e420 con 0x7f5ff40f51a0 2023-11-15T04:10:23.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.583+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff4045ad0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.583+0000 7f5fea7fc700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 0x7f5fe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:23.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.583+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ff00410b0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.583+0000 7f5ff9078700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 0x7f5fe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:23.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.584+0000 7f5ff9078700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 0x7f5fe00323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5fe40096a0 tx=0x7f5fe4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:23.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.588+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ff00095f0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.795+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ff410c2a0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.796+0000 7f5fea7fc700 1 -- 172.21.15.188:0/2069213650 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ff002c2d0 con 0x7f5ff40f51a0 2023-11-15T04:10:23.797 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:23.797 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:23.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 msgr2=0x7f5fe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:23.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 0x7f5fe00323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5fe40096a0 tx=0x7f5fe4006b40).stop 2023-11-15T04:10:23.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 msgr2=0x7f5ff410b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5ff0009830 tx=0x7f5ff000b040).stop 2023-11-15T04:10:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 shutdown_connections 2023-11-15T04:10:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fe002ff10 0x7f5fe00323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 --2- 172.21.15.188:0/2069213650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ff40f51a0 0x7f5ff410b5f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:23.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 >> 172.21.15.188:0/2069213650 conn(0x7f5ff40f09a0 msgr2=0x7f5ff40f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:23.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 shutdown_connections 2023-11-15T04:10:23.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:23.797+0000 7f5ffa87b700 1 -- 172.21.15.188:0/2069213650 wait complete. 2023-11-15T04:10:23.800 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:24.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:23.797140+0000 mon.smithi038 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.188:0/2069213650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:25.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:24 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:24.060232+0000 mgr.smithi038.bnbogz (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:25.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:25.431 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:26.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.652+0000 7f5e28179700 1 -- 172.21.15.188:0/1879869061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 msgr2=0x7f5e200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:26.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.652+0000 7f5e28179700 1 --2- 172.21.15.188:0/1879869061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e200f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5e10009fb0 tx=0x7f5e10009b30).stop 2023-11-15T04:10:26.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.652+0000 7f5e28179700 1 -- 172.21.15.188:0/1879869061 shutdown_connections 2023-11-15T04:10:26.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.652+0000 7f5e28179700 1 --2- 172.21.15.188:0/1879869061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e200f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:26.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.652+0000 7f5e28179700 1 -- 172.21.15.188:0/1879869061 >> 172.21.15.188:0/1879869061 conn(0x7f5e200f1c70 msgr2=0x7f5e200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:26.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 -- 172.21.15.188:0/1879869061 shutdown_connections 2023-11-15T04:10:26.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 -- 172.21.15.188:0/1879869061 wait complete. 2023-11-15T04:10:26.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 Processor -- start 2023-11-15T04:10:26.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 -- start start 2023-11-15T04:10:26.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:26.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e28179700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20106c20 con 0x7f5e200f6510 2023-11-15T04:10:26.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e25f15700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:26.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e25f15700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58270/0 (socket says 172.21.15.188:58270) 2023-11-15T04:10:26.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.653+0000 7f5e25f15700 1 -- 172.21.15.188:0/2121171916 learned_addr learned my addr 172.21.15.188:0/2121171916 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:26.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e25f15700 1 -- 172.21.15.188:0/2121171916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10005040 con 0x7f5e200f6510 2023-11-15T04:10:26.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e25f15700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5e10014000 tx=0x7f5e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:26.659 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e100095f0 con 0x7f5e200f6510 2023-11-15T04:10:26.659 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e20106e20 con 0x7f5e200f6510 2023-11-15T04:10:26.659 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e10007e70 con 0x7f5e200f6510 2023-11-15T04:10:26.660 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e10014ea0 con 0x7f5e200f6510 2023-11-15T04:10:26.660 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.654+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e201072b0 con 0x7f5e200f6510 2023-11-15T04:10:26.662 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.655+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e10025070 con 0x7f5e200f6510 2023-11-15T04:10:26.663 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.655+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e20045ad0 con 0x7f5e200f6510 2023-11-15T04:10:26.663 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.655+0000 7f5e167fc700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 0x7f5e0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:26.664 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.655+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e10040980 con 0x7f5e200f6510 2023-11-15T04:10:26.664 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.655+0000 7f5e25714700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 0x7f5e0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:26.664 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.656+0000 7f5e25714700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 0x7f5e0c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5e1c0096a0 tx=0x7f5e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:26.665 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.660+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e100079c0 con 0x7f5e200f6510 2023-11-15T04:10:26.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.864+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e200fbf60 con 0x7f5e200f6510 2023-11-15T04:10:26.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.865+0000 7f5e167fc700 1 -- 172.21.15.188:0/2121171916 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e1000f300 con 0x7f5e200f6510 2023-11-15T04:10:26.866 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:26.866 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:26.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 msgr2=0x7f5e0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:26.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 0x7f5e0c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5e1c0096a0 tx=0x7f5e1c006b40).stop 2023-11-15T04:10:26.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 msgr2=0x7f5e20106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:26.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5e10014000 tx=0x7f5e1000b040).stop 2023-11-15T04:10:26.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 shutdown_connections 2023-11-15T04:10:26.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e0c02ff60 0x7f5e0c032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:26.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 --2- 172.21.15.188:0/2121171916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e200f6510 0x7f5e20106730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:26.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 >> 172.21.15.188:0/2121171916 conn(0x7f5e200f1c70 msgr2=0x7f5e200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:26.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 shutdown_connections 2023-11-15T04:10:26.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:26.866+0000 7f5e28179700 1 -- 172.21.15.188:0/2121171916 wait complete. 2023-11-15T04:10:26.869 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:27.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:26.060574+0000 mgr.smithi038.bnbogz (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:27.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:26.865983+0000 mon.smithi038 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.188:0/2121171916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:28.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:28.269 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:29.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.477+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/1738397950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 msgr2=0x7fb59c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:29.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.477+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/1738397950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c0f9150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb58c009fb0 tx=0x7fb58c009b30).stop 2023-11-15T04:10:29.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.477+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/1738397950 shutdown_connections 2023-11-15T04:10:29.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.477+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/1738397950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c0f9150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:29.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.477+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/1738397950 >> 172.21.15.188:0/1738397950 conn(0x7fb59c0f0b70 msgr2=0x7fb59c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:29.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/1738397950 shutdown_connections 2023-11-15T04:10:29.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/1738397950 wait complete. 2023-11-15T04:10:29.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 Processor -- start 2023-11-15T04:10:29.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 -- start start 2023-11-15T04:10:29.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a3bb0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c108de0 con 0x7fb59c060220 2023-11-15T04:10:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a194c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a194c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:48616/0 (socket says 172.21.15.188:48616) 2023-11-15T04:10:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.478+0000 7fb5a194c700 1 -- 172.21.15.188:0/3964511010 learned_addr learned my addr 172.21.15.188:0/3964511010 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5a194c700 1 -- 172.21.15.188:0/3964511010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c005040 con 0x7fb59c060220 2023-11-15T04:10:29.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5a194c700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb58c013510 tx=0x7fb58c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:29.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58c009320 con 0x7fb59c060220 2023-11-15T04:10:29.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c108fe0 con 0x7fb59c060220 2023-11-15T04:10:29.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c109470 con 0x7fb59c060220 2023-11-15T04:10:29.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb58c013f00 con 0x7fb59c060220 2023-11-15T04:10:29.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.479+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58c014d90 con 0x7fb59c060220 2023-11-15T04:10:29.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.480+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb58c0263a0 con 0x7fb59c060220 2023-11-15T04:10:29.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.480+0000 7fb5927fc700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:29.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.480+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb58c041030 con 0x7fb59c060220 2023-11-15T04:10:29.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.480+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c045ad0 con 0x7fb59c060220 2023-11-15T04:10:29.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.480+0000 7fb5a114b700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:29.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.481+0000 7fb5a114b700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 0x7fb588032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb5980096a0 tx=0x7fb598006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:29.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.485+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb58c01d380 con 0x7fb59c060220 2023-11-15T04:10:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:28.060891+0000 mgr.smithi038.bnbogz (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:29.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.689+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb59c0fc040 con 0x7fb59c060220 2023-11-15T04:10:29.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.690+0000 7fb5927fc700 1 -- 172.21.15.188:0/3964511010 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb58c02b020 con 0x7fb59c060220 2023-11-15T04:10:29.691 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:29.691 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:29.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 msgr2=0x7fb588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 0x7fb588032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb5980096a0 tx=0x7fb598006b40).stop 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 msgr2=0x7fb59c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb58c013510 tx=0x7fb58c00b040).stop 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 shutdown_connections 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:29.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 --2- 172.21.15.188:0/3964511010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb59c060220 0x7fb59c1088f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:29.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 >> 172.21.15.188:0/3964511010 conn(0x7fb59c0f0b70 msgr2=0x7fb59c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:29.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 shutdown_connections 2023-11-15T04:10:29.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:29.692+0000 7fb5a3bb0700 1 -- 172.21.15.188:0/3964511010 wait complete. 2023-11-15T04:10:29.695 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:30.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:29.691570+0000 mon.smithi038 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.188:0/3964511010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:31.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:31.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:30.061241+0000 mgr.smithi038.bnbogz (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:32.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 -- 172.21.15.188:0/4089316716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 msgr2=0x7febd40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:32.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 --2- 172.21.15.188:0/4089316716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd40f9150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7febc4009fb0 tx=0x7febc4009b30).stop 2023-11-15T04:10:32.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 -- 172.21.15.188:0/4089316716 shutdown_connections 2023-11-15T04:10:32.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 --2- 172.21.15.188:0/4089316716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd40f9150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:32.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 -- 172.21.15.188:0/4089316716 >> 172.21.15.188:0/4089316716 conn(0x7febd40f0b70 msgr2=0x7febd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:32.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 -- 172.21.15.188:0/4089316716 shutdown_connections 2023-11-15T04:10:32.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.323+0000 7febd90e8700 1 -- 172.21.15.188:0/4089316716 wait complete. 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd90e8700 1 Processor -- start 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd90e8700 1 -- start start 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd90e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd90e8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4106ba0 con 0x7febd4060220 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:32.327 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:48640/0 (socket says 172.21.15.188:48640) 2023-11-15T04:10:32.328 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.324+0000 7febd2d9d700 1 -- 172.21.15.188:0/3059275738 learned_addr learned my addr 172.21.15.188:0/3059275738 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:32.328 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febd2d9d700 1 -- 172.21.15.188:0/3059275738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4005040 con 0x7febd4060220 2023-11-15T04:10:32.329 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febd2d9d700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7febc4013510 tx=0x7febc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:32.329 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc4009320 con 0x7febd4060220 2023-11-15T04:10:32.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd4106da0 con 0x7febd4060220 2023-11-15T04:10:32.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febc4013f00 con 0x7febd4060220 2023-11-15T04:10:32.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc4014dc0 con 0x7febd4060220 2023-11-15T04:10:32.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.325+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd4107230 con 0x7febd4060220 2023-11-15T04:10:32.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.326+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febc40095f0 con 0x7febd4060220 2023-11-15T04:10:32.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.326+0000 7febbb7fe700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 0x7febbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:32.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.326+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd4045ad0 con 0x7febd4060220 2023-11-15T04:10:32.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.326+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febc40410c0 con 0x7febd4060220 2023-11-15T04:10:32.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.326+0000 7febd259c700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 0x7febbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:32.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.327+0000 7febd259c700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 0x7febbc032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7febc80096a0 tx=0x7febc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:32.332 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.331+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febc401c790 con 0x7febd4060220 2023-11-15T04:10:32.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.535+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febd41080e0 con 0x7febd4060220 2023-11-15T04:10:32.537 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.536+0000 7febbb7fe700 1 -- 172.21.15.188:0/3059275738 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febc4010230 con 0x7febd4060220 2023-11-15T04:10:32.537 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:32.538 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:32.538 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 msgr2=0x7febbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:32.539 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 0x7febbc032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7febc80096a0 tx=0x7febc8006b40).stop 2023-11-15T04:10:32.539 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 msgr2=0x7febd41066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:32.539 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7febc4013510 tx=0x7febc400b040).stop 2023-11-15T04:10:32.539 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 shutdown_connections 2023-11-15T04:10:32.540 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febbc02ff60 0x7febbc032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:32.540 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 --2- 172.21.15.188:0/3059275738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4060220 0x7febd41066b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:32.540 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 >> 172.21.15.188:0/3059275738 conn(0x7febd40f0b70 msgr2=0x7febd40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:32.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 shutdown_connections 2023-11-15T04:10:32.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:32.538+0000 7febd90e8700 1 -- 172.21.15.188:0/3059275738 wait complete. 2023-11-15T04:10:32.541 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:32.061538+0000 mgr.smithi038.bnbogz (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:32.537595+0000 mon.smithi038 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.188:0/3059275738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:33.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:33.954 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:34.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:34.061869+0000 mgr.smithi038.bnbogz (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:35.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 -- 172.21.15.188:0/184199329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 msgr2=0x7f26440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:35.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 --2- 172.21.15.188:0/184199329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f26440f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f2638009fb0 tx=0x7f2638009b30).stop 2023-11-15T04:10:35.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 -- 172.21.15.188:0/184199329 shutdown_connections 2023-11-15T04:10:35.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 --2- 172.21.15.188:0/184199329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f26440f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:35.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 -- 172.21.15.188:0/184199329 >> 172.21.15.188:0/184199329 conn(0x7f26440f0b70 msgr2=0x7f26440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:35.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 -- 172.21.15.188:0/184199329 shutdown_connections 2023-11-15T04:10:35.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.116+0000 7f264a0b2700 1 -- 172.21.15.188:0/184199329 wait complete. 2023-11-15T04:10:35.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f264a0b2700 1 Processor -- start 2023-11-15T04:10:35.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f264a0b2700 1 -- start start 2023-11-15T04:10:35.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f264a0b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:35.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f264a0b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2644060240 con 0x7f26440f5410 2023-11-15T04:10:35.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f26437fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:35.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f26437fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:48664/0 (socket says 172.21.15.188:48664) 2023-11-15T04:10:35.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.117+0000 7f26437fe700 1 -- 172.21.15.188:0/3300051072 learned_addr learned my addr 172.21.15.188:0/3300051072 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:35.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f26437fe700 1 -- 172.21.15.188:0/3300051072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2638005040 con 0x7f26440f5410 2023-11-15T04:10:35.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f26437fe700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f2638014000 tx=0x7f263800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:35.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26380095f0 con 0x7f26440f5410 2023-11-15T04:10:35.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2644060440 con 0x7f26440f5410 2023-11-15T04:10:35.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2638007e70 con 0x7f26440f5410 2023-11-15T04:10:35.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2638014ea0 con 0x7f26440f5410 2023-11-15T04:10:35.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.118+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2644109ae0 con 0x7f26440f5410 2023-11-15T04:10:35.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.119+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2638025070 con 0x7f26440f5410 2023-11-15T04:10:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.119+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2644045ad0 con 0x7f26440f5410 2023-11-15T04:10:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.119+0000 7f262bfff700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 0x7f262c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.119+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26380409f0 con 0x7f26440f5410 2023-11-15T04:10:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.119+0000 7f2642ffd700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 0x7f262c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:35.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.120+0000 7f2642ffd700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 0x7f262c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f26340096a0 tx=0x7f2634006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:35.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.124+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f263800f4e0 con 0x7f26440f5410 2023-11-15T04:10:35.330 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.330+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26440fae60 con 0x7f26440f5410 2023-11-15T04:10:35.331 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.330+0000 7f262bfff700 1 -- 172.21.15.188:0/3300051072 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f263800f300 con 0x7f26440f5410 2023-11-15T04:10:35.332 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:35.332 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:35.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 msgr2=0x7f262c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:35.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 0x7f262c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f26340096a0 tx=0x7f2634006b40).stop 2023-11-15T04:10:35.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 msgr2=0x7f2644109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:35.333 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f2638014000 tx=0x7f263800b040).stop 2023-11-15T04:10:35.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 shutdown_connections 2023-11-15T04:10:35.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f262c02ff60 0x7f262c032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:35.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 --2- 172.21.15.188:0/3300051072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26440f5410 0x7f2644109430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:35.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 >> 172.21.15.188:0/3300051072 conn(0x7f26440f0b70 msgr2=0x7f26440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:35.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 shutdown_connections 2023-11-15T04:10:35.335 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:35.332+0000 7f264a0b2700 1 -- 172.21.15.188:0/3300051072 wait complete. 2023-11-15T04:10:35.335 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:35.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:35.331888+0000 mon.smithi038 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.188:0/3300051072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:36.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:36.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:36.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:36 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:36.062220+0000 mgr.smithi038.bnbogz (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:37.923 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 -- 172.21.15.188:0/3166093509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 msgr2=0x7f0b000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:37.923 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 --2- 172.21.15.188:0/3166093509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0af4009fb0 tx=0x7f0af4009b30).stop 2023-11-15T04:10:37.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 -- 172.21.15.188:0/3166093509 shutdown_connections 2023-11-15T04:10:37.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 --2- 172.21.15.188:0/3166093509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:37.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 -- 172.21.15.188:0/3166093509 >> 172.21.15.188:0/3166093509 conn(0x7f0b000f0b70 msgr2=0x7f0b000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:37.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 -- 172.21.15.188:0/3166093509 shutdown_connections 2023-11-15T04:10:37.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.922+0000 7f0b05868700 1 -- 172.21.15.188:0/3166093509 wait complete. 2023-11-15T04:10:37.926 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0b05868700 1 Processor -- start 2023-11-15T04:10:37.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0b05868700 1 -- start start 2023-11-15T04:10:37.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0b05868700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:37.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0b05868700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b000fb820 con 0x7f0b000f5410 2023-11-15T04:10:37.927 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0afeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0afeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:48686/0 (socket says 172.21.15.188:48686) 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.923+0000 7f0afeffd700 1 -- 172.21.15.188:0/1504888598 learned_addr learned my addr 172.21.15.188:0/1504888598 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0afeffd700 1 -- 172.21.15.188:0/1504888598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4005040 con 0x7f0b000f5410 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0afeffd700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0af4013510 tx=0x7f0af400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0af4009320 con 0x7f0b000f5410 2023-11-15T04:10:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0af4013f00 con 0x7f0b000f5410 2023-11-15T04:10:37.929 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b000fb9c0 con 0x7f0b000f5410 2023-11-15T04:10:37.929 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0af4014d90 con 0x7f0b000f5410 2023-11-15T04:10:37.929 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.924+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b000fbe50 con 0x7f0b000f5410 2023-11-15T04:10:37.929 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0af4014420 con 0x7f0b000f5410 2023-11-15T04:10:37.929 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b00059100 con 0x7f0b000f5410 2023-11-15T04:10:37.930 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0ae77fe700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 0x7f0ae8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:37.930 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0af4041070 con 0x7f0b000f5410 2023-11-15T04:10:37.930 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0afe7fc700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 0x7f0ae8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:37.930 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.925+0000 7f0afe7fc700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 0x7f0ae8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0af00096a0 tx=0x7f0af0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:37.931 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:37.930+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0af40095f0 con 0x7f0b000f5410 2023-11-15T04:10:38.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.137+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b000fae60 con 0x7f0b000f5410 2023-11-15T04:10:38.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.137+0000 7f0ae77fe700 1 -- 172.21.15.188:0/1504888598 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0af401db70 con 0x7f0b000f5410 2023-11-15T04:10:38.139 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:38.139 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:38.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 msgr2=0x7f0ae8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:38.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 0x7f0ae8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0af00096a0 tx=0x7f0af0006b40).stop 2023-11-15T04:10:38.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 msgr2=0x7f0b000fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:38.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0af4013510 tx=0x7f0af400b040).stop 2023-11-15T04:10:38.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 shutdown_connections 2023-11-15T04:10:38.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ae802ff60 0x7f0ae8032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:38.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 --2- 172.21.15.188:0/1504888598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b000f5410 0x7f0b000fd3a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:38.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 >> 172.21.15.188:0/1504888598 conn(0x7f0b000f0b70 msgr2=0x7f0b000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:38.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 shutdown_connections 2023-11-15T04:10:38.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:38.139+0000 7f0b05868700 1 -- 172.21.15.188:0/1504888598 wait complete. 2023-11-15T04:10:38.142 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:39.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:39.526 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:38.062469+0000 mgr.smithi038.bnbogz (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:38.138981+0000 mon.smithi038 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.188:0/1504888598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:40.751 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.749+0000 7fec1f451700 1 -- 172.21.15.188:0/2206605608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 msgr2=0x7fec180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:40.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.749+0000 7fec1f451700 1 --2- 172.21.15.188:0/2206605608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec180f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fec0c009fb0 tx=0x7fec0c009b30).stop 2023-11-15T04:10:40.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.749+0000 7fec1f451700 1 -- 172.21.15.188:0/2206605608 shutdown_connections 2023-11-15T04:10:40.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.749+0000 7fec1f451700 1 --2- 172.21.15.188:0/2206605608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec180f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:40.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.749+0000 7fec1f451700 1 -- 172.21.15.188:0/2206605608 >> 172.21.15.188:0/2206605608 conn(0x7fec180f0b70 msgr2=0x7fec180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:40.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 -- 172.21.15.188:0/2206605608 shutdown_connections 2023-11-15T04:10:40.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 -- 172.21.15.188:0/2206605608 wait complete. 2023-11-15T04:10:40.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 Processor -- start 2023-11-15T04:10:40.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 -- start start 2023-11-15T04:10:40.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:40.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1f451700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18105b20 con 0x7fec180f5410 2023-11-15T04:10:40.754 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.750+0000 7fec1d1ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:40.754 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1d1ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42188/0 (socket says 172.21.15.188:42188) 2023-11-15T04:10:40.754 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1d1ed700 1 -- 172.21.15.188:0/216831295 learned_addr learned my addr 172.21.15.188:0/216831295 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:40.754 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1d1ed700 1 -- 172.21.15.188:0/216831295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c005040 con 0x7fec180f5410 2023-11-15T04:10:40.754 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1d1ed700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fec0c013510 tx=0x7fec0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:40.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c009320 con 0x7fec180f5410 2023-11-15T04:10:40.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec18105d20 con 0x7fec180f5410 2023-11-15T04:10:40.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec0c013f00 con 0x7fec180f5410 2023-11-15T04:10:40.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c014d90 con 0x7fec180f5410 2023-11-15T04:10:40.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.751+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec181061b0 con 0x7fec180f5410 2023-11-15T04:10:40.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.752+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec0c014420 con 0x7fec180f5410 2023-11-15T04:10:40.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.752+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18045ad0 con 0x7fec180f5410 2023-11-15T04:10:40.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.752+0000 7fec09ffb700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:40.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.752+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec0c0410c0 con 0x7fec180f5410 2023-11-15T04:10:40.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.753+0000 7fec1c9ec700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:40.758 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.753+0000 7fec1c9ec700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fec140096a0 tx=0x7fec14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:40.758 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.757+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0c0095f0 con 0x7fec180f5410 2023-11-15T04:10:40.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.963+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec180fae60 con 0x7fec180f5410 2023-11-15T04:10:40.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.963+0000 7fec09ffb700 1 -- 172.21.15.188:0/216831295 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec0c01db70 con 0x7fec180f5410 2023-11-15T04:10:40.964 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:40.965 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:40.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 msgr2=0x7fec04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:40.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fec140096a0 tx=0x7fec14006b40).stop 2023-11-15T04:10:40.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 msgr2=0x7fec18105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:40.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fec0c013510 tx=0x7fec0c00b040).stop 2023-11-15T04:10:40.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 shutdown_connections 2023-11-15T04:10:40.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:40.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 --2- 172.21.15.188:0/216831295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec180f5410 0x7fec18105630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:40.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 >> 172.21.15.188:0/216831295 conn(0x7fec180f0b70 msgr2=0x7fec180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:40.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 shutdown_connections 2023-11-15T04:10:40.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:40.965+0000 7fec1f451700 1 -- 172.21.15.188:0/216831295 wait complete. 2023-11-15T04:10:40.968 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:40.062794+0000 mgr.smithi038.bnbogz (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:41.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:40.964631+0000 mon.smithi038 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.188:0/216831295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:42.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:42.371 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:42.063088+0000 mgr.smithi038.bnbogz (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:43.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.510+0000 7f8c63841700 1 -- 172.21.15.188:0/2657035591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 msgr2=0x7f8c5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:43.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.510+0000 7f8c63841700 1 --2- 172.21.15.188:0/2657035591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8c58009fb0 tx=0x7f8c58009b30).stop 2023-11-15T04:10:43.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.510+0000 7f8c63841700 1 -- 172.21.15.188:0/2657035591 shutdown_connections 2023-11-15T04:10:43.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.510+0000 7f8c63841700 1 --2- 172.21.15.188:0/2657035591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:43.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.510+0000 7f8c63841700 1 -- 172.21.15.188:0/2657035591 >> 172.21.15.188:0/2657035591 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 -- 172.21.15.188:0/2657035591 shutdown_connections 2023-11-15T04:10:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 -- 172.21.15.188:0/2657035591 wait complete. 2023-11-15T04:10:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 Processor -- start 2023-11-15T04:10:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 -- start start 2023-11-15T04:10:43.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:43.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.511+0000 7f8c63841700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c105b20 con 0x7f8c5c0f5410 2023-11-15T04:10:43.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c615dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:43.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c615dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42204/0 (socket says 172.21.15.188:42204) 2023-11-15T04:10:43.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c615dd700 1 -- 172.21.15.188:0/521303850 learned_addr learned my addr 172.21.15.188:0/521303850 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:43.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c615dd700 1 -- 172.21.15.188:0/521303850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58005040 con 0x7f8c5c0f5410 2023-11-15T04:10:43.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c615dd700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8c58013510 tx=0x7f8c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:43.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c58009320 con 0x7f8c5c0f5410 2023-11-15T04:10:43.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c105d20 con 0x7f8c5c0f5410 2023-11-15T04:10:43.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c58013f00 con 0x7f8c5c0f5410 2023-11-15T04:10:43.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.512+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c58014d90 con 0x7f8c5c0f5410 2023-11-15T04:10:43.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.513+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c1061b0 con 0x7f8c5c0f5410 2023-11-15T04:10:43.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.513+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c58014420 con 0x7f8c5c0f5410 2023-11-15T04:10:43.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.513+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c045ad0 con 0x7f8c5c0f5410 2023-11-15T04:10:43.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.513+0000 7f8c51ffb700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:43.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.514+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c58042e70 con 0x7f8c5c0f5410 2023-11-15T04:10:43.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.514+0000 7f8c60ddc700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:43.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.514+0000 7f8c60ddc700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 0x7f8c48032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c4c0096a0 tx=0x7f8c4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:43.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.518+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c5801fb70 con 0x7f8c5c0f5410 2023-11-15T04:10:43.727 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.726+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c5c0faf50 con 0x7f8c5c0f5410 2023-11-15T04:10:43.728 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.727+0000 7f8c51ffb700 1 -- 172.21.15.188:0/521303850 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c5801e730 con 0x7f8c5c0f5410 2023-11-15T04:10:43.728 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:43.729 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:43.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 msgr2=0x7f8c48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:43.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 0x7f8c48032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c4c0096a0 tx=0x7f8c4c006b40).stop 2023-11-15T04:10:43.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 msgr2=0x7f8c5c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:43.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8c58013510 tx=0x7f8c5800b040).stop 2023-11-15T04:10:43.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 shutdown_connections 2023-11-15T04:10:43.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:43.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 --2- 172.21.15.188:0/521303850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c105630 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:43.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 >> 172.21.15.188:0/521303850 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:43.732 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 shutdown_connections 2023-11-15T04:10:43.732 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:43.729+0000 7f8c63841700 1 -- 172.21.15.188:0/521303850 wait complete. 2023-11-15T04:10:43.732 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:44.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:43 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:43.728448+0000 mon.smithi038 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.188:0/521303850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:45.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:44 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:44.063371+0000 mgr.smithi038.bnbogz (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:45.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:45.114 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:46.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 -- 172.21.15.188:0/648650518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 msgr2=0x7ff4cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:46.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 --2- 172.21.15.188:0/648650518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc0f90f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff4bc009fb0 tx=0x7ff4bc009b30).stop 2023-11-15T04:10:46.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 -- 172.21.15.188:0/648650518 shutdown_connections 2023-11-15T04:10:46.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 --2- 172.21.15.188:0/648650518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc0f90f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:46.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 -- 172.21.15.188:0/648650518 >> 172.21.15.188:0/648650518 conn(0x7ff4cc0f0b70 msgr2=0x7ff4cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:46.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 -- 172.21.15.188:0/648650518 shutdown_connections 2023-11-15T04:10:46.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.280+0000 7ff4d3848700 1 -- 172.21.15.188:0/648650518 wait complete. 2023-11-15T04:10:46.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d3848700 1 Processor -- start 2023-11-15T04:10:46.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d3848700 1 -- start start 2023-11-15T04:10:46.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d3848700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:46.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d3848700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc106bb0 con 0x7ff4cc060280 2023-11-15T04:10:46.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d15e4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:46.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d15e4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42234/0 (socket says 172.21.15.188:42234) 2023-11-15T04:10:46.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d15e4700 1 -- 172.21.15.188:0/2339279666 learned_addr learned my addr 172.21.15.188:0/2339279666 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:46.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.281+0000 7ff4d15e4700 1 -- 172.21.15.188:0/2339279666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc005040 con 0x7ff4cc060280 2023-11-15T04:10:46.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4d15e4700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff4bc013510 tx=0x7ff4bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:46.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4bc009320 con 0x7ff4cc060280 2023-11-15T04:10:46.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc106db0 con 0x7ff4cc060280 2023-11-15T04:10:46.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4bc013f00 con 0x7ff4cc060280 2023-11-15T04:10:46.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4bc014dc0 con 0x7ff4cc060280 2023-11-15T04:10:46.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.282+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc107240 con 0x7ff4cc060280 2023-11-15T04:10:46.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.283+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4bc014420 con 0x7ff4cc060280 2023-11-15T04:10:46.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.283+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc045ad0 con 0x7ff4cc060280 2023-11-15T04:10:46.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.283+0000 7ff4c1ffb700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:46.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.283+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4bc041020 con 0x7ff4cc060280 2023-11-15T04:10:46.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.283+0000 7ff4d0de3700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:46.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.284+0000 7ff4d0de3700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 0x7ff4b8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff4c80096a0 tx=0x7ff4c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:46.289 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.288+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4bc0095f0 con 0x7ff4cc060280 2023-11-15T04:10:46.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.495+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4cc108090 con 0x7ff4cc060280 2023-11-15T04:10:46.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.495+0000 7ff4c1ffb700 1 -- 172.21.15.188:0/2339279666 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4bc01cb10 con 0x7ff4cc060280 2023-11-15T04:10:46.499 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:46.499 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:46.500 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 msgr2=0x7ff4b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:46.500 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 0x7ff4b8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff4c80096a0 tx=0x7ff4c8006b40).stop 2023-11-15T04:10:46.500 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 msgr2=0x7ff4cc1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:46.500 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff4bc013510 tx=0x7ff4bc00b040).stop 2023-11-15T04:10:46.501 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 shutdown_connections 2023-11-15T04:10:46.501 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:46.501 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 --2- 172.21.15.188:0/2339279666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc060280 0x7ff4cc1066c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:46.502 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 >> 172.21.15.188:0/2339279666 conn(0x7ff4cc0f0b70 msgr2=0x7ff4cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:46.502 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 shutdown_connections 2023-11-15T04:10:46.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:46.497+0000 7ff4d3848700 1 -- 172.21.15.188:0/2339279666 wait complete. 2023-11-15T04:10:46.503 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:46.063663+0000 mgr.smithi038.bnbogz (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:46.496891+0000 mon.smithi038 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.188:0/2339279666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:47.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:47.824 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:49.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 -- 172.21.15.188:0/3915855314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 msgr2=0x7febe00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:49.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 --2- 172.21.15.188:0/3915855314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe00f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7febc8009fb0 tx=0x7febc8009b30).stop 2023-11-15T04:10:49.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 -- 172.21.15.188:0/3915855314 shutdown_connections 2023-11-15T04:10:49.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 --2- 172.21.15.188:0/3915855314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe00f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:49.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 -- 172.21.15.188:0/3915855314 >> 172.21.15.188:0/3915855314 conn(0x7febe00f0b70 msgr2=0x7febe00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:49.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 -- 172.21.15.188:0/3915855314 shutdown_connections 2023-11-15T04:10:49.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.048+0000 7febe495d700 1 -- 172.21.15.188:0/3915855314 wait complete. 2023-11-15T04:10:49.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febe495d700 1 Processor -- start 2023-11-15T04:10:49.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febe495d700 1 -- start start 2023-11-15T04:10:49.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febe495d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:49.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febe495d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe0106c20 con 0x7febe00f5410 2023-11-15T04:10:49.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febde59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:49.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febde59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42700/0 (socket says 172.21.15.188:42700) 2023-11-15T04:10:49.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.049+0000 7febde59c700 1 -- 172.21.15.188:0/1916509905 learned_addr learned my addr 172.21.15.188:0/1916509905 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:49.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febde59c700 1 -- 172.21.15.188:0/1916509905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8005040 con 0x7febe00f5410 2023-11-15T04:10:49.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febde59c700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7febc8014000 tx=0x7febc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:49.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc80095f0 con 0x7febe00f5410 2023-11-15T04:10:49.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febc8007e70 con 0x7febe00f5410 2023-11-15T04:10:49.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe0106e20 con 0x7febe00f5410 2023-11-15T04:10:49.056 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8014ea0 con 0x7febe00f5410 2023-11-15T04:10:49.056 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.050+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe01072b0 con 0x7febe00f5410 2023-11-15T04:10:49.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.051+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febc8025070 con 0x7febe00f5410 2023-11-15T04:10:49.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.051+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe0045ad0 con 0x7febe00f5410 2023-11-15T04:10:49.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.051+0000 7febd6ffd700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 0x7febcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:49.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.051+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febc8040970 con 0x7febe00f5410 2023-11-15T04:10:49.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.051+0000 7febddd9b700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 0x7febcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:49.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.052+0000 7febddd9b700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 0x7febcc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7febd00096a0 tx=0x7febd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:49.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.056+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febc80079c0 con 0x7febe00f5410 2023-11-15T04:10:49.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.261+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febe00fae60 con 0x7febe00f5410 2023-11-15T04:10:49.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.261+0000 7febd6ffd700 1 -- 172.21.15.188:0/1916509905 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febc800f300 con 0x7febe00f5410 2023-11-15T04:10:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:49.263 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:49.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 msgr2=0x7febcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:49.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 0x7febcc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7febd00096a0 tx=0x7febd0006b40).stop 2023-11-15T04:10:49.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 msgr2=0x7febe0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7febc8014000 tx=0x7febc800b040).stop 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 shutdown_connections 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7febcc02ff60 0x7febcc032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 --2- 172.21.15.188:0/1916509905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febe00f5410 0x7febe0106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 >> 172.21.15.188:0/1916509905 conn(0x7febe00f0b70 msgr2=0x7febe00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:49.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 shutdown_connections 2023-11-15T04:10:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:49.263+0000 7febe495d700 1 -- 172.21.15.188:0/1916509905 wait complete. 2023-11-15T04:10:49.266 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:48.063994+0000 mgr.smithi038.bnbogz (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:49.262734+0000 mon.smithi038 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.188:0/1916509905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:50.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:50.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:50.064256+0000 mgr.smithi038.bnbogz (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:50.208449+0000 mon.smithi038 (mon.0) 151 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:10:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:50.211189+0000 mon.smithi038 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:10:51.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- 172.21.15.188:0/1936066101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 msgr2=0x7f60240f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:51.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 --2- 172.21.15.188:0/1936066101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f60240f4be0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6014009fb0 tx=0x7f6014009b30).stop 2023-11-15T04:10:51.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- 172.21.15.188:0/1936066101 shutdown_connections 2023-11-15T04:10:51.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 --2- 172.21.15.188:0/1936066101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f60240f4be0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:51.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- 172.21.15.188:0/1936066101 >> 172.21.15.188:0/1936066101 conn(0x7f60240f02e0 msgr2=0x7f60240f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:51.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- 172.21.15.188:0/1936066101 shutdown_connections 2023-11-15T04:10:51.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- 172.21.15.188:0/1936066101 wait complete. 2023-11-15T04:10:51.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 Processor -- start 2023-11-15T04:10:51.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.905+0000 7f6028d38700 1 -- start start 2023-11-15T04:10:51.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6028d38700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:51.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6028d38700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6024106bf0 con 0x7f60240f47c0 2023-11-15T04:10:51.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f602259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f602259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42714/0 (socket says 172.21.15.188:42714) 2023-11-15T04:10:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f602259c700 1 -- 172.21.15.188:0/1557630450 learned_addr learned my addr 172.21.15.188:0/1557630450 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f602259c700 1 -- 172.21.15.188:0/1557630450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014005040 con 0x7f60240f47c0 2023-11-15T04:10:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f602259c700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6014013510 tx=0x7f601400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:51.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6014009320 con 0x7f60240f47c0 2023-11-15T04:10:51.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6024106df0 con 0x7f60240f47c0 2023-11-15T04:10:51.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6014013f00 con 0x7f60240f47c0 2023-11-15T04:10:51.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.906+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6024107280 con 0x7f60240f47c0 2023-11-15T04:10:51.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.907+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6014014d90 con 0x7f60240f47c0 2023-11-15T04:10:51.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.907+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6024045ad0 con 0x7f60240f47c0 2023-11-15T04:10:51.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.907+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60140095f0 con 0x7f60240f47c0 2023-11-15T04:10:51.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.907+0000 7f6012ffd700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 0x7f600c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:51.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.908+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6014040fd0 con 0x7f60240f47c0 2023-11-15T04:10:51.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.908+0000 7f6021d9b700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 0x7f600c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:51.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.908+0000 7f6021d9b700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 0x7f600c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f60180096a0 tx=0x7f6018006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:51.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:51.912+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6014014420 con 0x7f60240f47c0 2023-11-15T04:10:52.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.118+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60241080d0 con 0x7f60240f47c0 2023-11-15T04:10:52.120 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.119+0000 7f6012ffd700 1 -- 172.21.15.188:0/1557630450 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6014010230 con 0x7f60240f47c0 2023-11-15T04:10:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:52.121 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:52.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 msgr2=0x7f600c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:52.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 0x7f600c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f60180096a0 tx=0x7f6018006b40).stop 2023-11-15T04:10:52.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 msgr2=0x7f6024106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:52.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6014013510 tx=0x7f601400b040).stop 2023-11-15T04:10:52.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 shutdown_connections 2023-11-15T04:10:52.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f600c02ff60 0x7f600c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:52.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 --2- 172.21.15.188:0/1557630450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60240f47c0 0x7f6024106700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:52.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 >> 172.21.15.188:0/1557630450 conn(0x7f60240f02e0 msgr2=0x7f60240f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:52.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 shutdown_connections 2023-11-15T04:10:52.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:52.120+0000 7f6028d38700 1 -- 172.21.15.188:0/1557630450 wait complete. 2023-11-15T04:10:52.125 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:52.064602+0000 mgr.smithi038.bnbogz (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:52.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:52.119993+0000 mon.smithi038 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.188:0/1557630450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:53.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:53.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:54.769 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 -- 172.21.15.188:0/3551269125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 msgr2=0x7f54240fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:54.769 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 --2- 172.21.15.188:0/3551269125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54240fb570 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f540c009fb0 tx=0x7f540c009b30).stop 2023-11-15T04:10:54.769 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 -- 172.21.15.188:0/3551269125 shutdown_connections 2023-11-15T04:10:54.769 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 --2- 172.21.15.188:0/3551269125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54240fb570 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:54.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 -- 172.21.15.188:0/3551269125 >> 172.21.15.188:0/3551269125 conn(0x7f54240f0b70 msgr2=0x7f54240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:54.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 -- 172.21.15.188:0/3551269125 shutdown_connections 2023-11-15T04:10:54.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.766+0000 7f5428949700 1 -- 172.21.15.188:0/3551269125 wait complete. 2023-11-15T04:10:54.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f5428949700 1 Processor -- start 2023-11-15T04:10:54.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f5428949700 1 -- start start 2023-11-15T04:10:54.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f5428949700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:54.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f5428949700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424108dc0 con 0x7f5424060200 2023-11-15T04:10:54.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f542259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:54.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f542259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42728/0 (socket says 172.21.15.188:42728) 2023-11-15T04:10:54.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.767+0000 7f542259c700 1 -- 172.21.15.188:0/3081810290 learned_addr learned my addr 172.21.15.188:0/3081810290 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:54.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f542259c700 1 -- 172.21.15.188:0/3081810290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c005040 con 0x7f5424060200 2023-11-15T04:10:54.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f542259c700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f540c013510 tx=0x7f540c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:54.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c009320 con 0x7f5424060200 2023-11-15T04:10:54.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424108fc0 con 0x7f5424060200 2023-11-15T04:10:54.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f540c013f00 con 0x7f5424060200 2023-11-15T04:10:54.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c014dc0 con 0x7f5424060200 2023-11-15T04:10:54.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.768+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5424109450 con 0x7f5424060200 2023-11-15T04:10:54.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.769+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f540c0095f0 con 0x7f5424060200 2023-11-15T04:10:54.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.769+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424045ad0 con 0x7f5424060200 2023-11-15T04:10:54.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.769+0000 7f541affd700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 0x7f5410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:54.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.769+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f540c041020 con 0x7f5424060200 2023-11-15T04:10:54.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.769+0000 7f5421d9b700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 0x7f5410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:54.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.770+0000 7f5421d9b700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 0x7f5410032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:54.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.774+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f540c014420 con 0x7f5424060200 2023-11-15T04:10:54.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.977+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f542410a2a0 con 0x7f5424060200 2023-11-15T04:10:54.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.978+0000 7f541affd700 1 -- 172.21.15.188:0/3081810290 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f540c01c920 con 0x7f5424060200 2023-11-15T04:10:54.979 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:54.980 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:54.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 msgr2=0x7f5410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:54.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 0x7f5410032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).stop 2023-11-15T04:10:54.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 msgr2=0x7f54241088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:54.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f540c013510 tx=0x7f540c00b040).stop 2023-11-15T04:10:54.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 shutdown_connections 2023-11-15T04:10:54.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f541002ff60 0x7f5410032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:54.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 --2- 172.21.15.188:0/3081810290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5424060200 0x7f54241088d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:54.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 >> 172.21.15.188:0/3081810290 conn(0x7f54240f0b70 msgr2=0x7f54240f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:54.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 shutdown_connections 2023-11-15T04:10:54.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:54.980+0000 7f5428949700 1 -- 172.21.15.188:0/3081810290 wait complete. 2023-11-15T04:10:54.984 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:54.064899+0000 mgr.smithi038.bnbogz (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:54.979460+0000 mon.smithi038 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.188:0/3081810290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:56.384 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:56.384 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:56.065179+0000 mgr.smithi038.bnbogz (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:10:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 -- 172.21.15.188:0/2857843246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 msgr2=0x7f9a9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/2857843246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c0f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9a84009fb0 tx=0x7f9a84009b30).stop 2023-11-15T04:10:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 -- 172.21.15.188:0/2857843246 shutdown_connections 2023-11-15T04:10:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/2857843246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c0f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 -- 172.21.15.188:0/2857843246 >> 172.21.15.188:0/2857843246 conn(0x7f9a9c0f0b70 msgr2=0x7f9a9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 -- 172.21.15.188:0/2857843246 shutdown_connections 2023-11-15T04:10:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.604+0000 7f9aa18a9700 1 -- 172.21.15.188:0/2857843246 wait complete. 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.605+0000 7f9aa18a9700 1 Processor -- start 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.605+0000 7f9aa18a9700 1 -- start start 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.605+0000 7f9aa18a9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.605+0000 7f9aa18a9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c105b20 con 0x7f9a9c0f5410 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a9affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a9affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42740/0 (socket says 172.21.15.188:42740) 2023-11-15T04:10:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a9affd700 1 -- 172.21.15.188:0/3027766490 learned_addr learned my addr 172.21.15.188:0/3027766490 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:10:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a9affd700 1 -- 172.21.15.188:0/3027766490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84005040 con 0x7f9a9c0f5410 2023-11-15T04:10:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a9affd700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9a84013510 tx=0x7f9a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a84009320 con 0x7f9a9c0f5410 2023-11-15T04:10:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c105d20 con 0x7f9a9c0f5410 2023-11-15T04:10:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.606+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a84013f00 con 0x7f9a9c0f5410 2023-11-15T04:10:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a84014d90 con 0x7f9a9c0f5410 2023-11-15T04:10:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c1061b0 con 0x7f9a9c0f5410 2023-11-15T04:10:57.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a84014420 con 0x7f9a9c0f5410 2023-11-15T04:10:57.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c045ad0 con 0x7f9a9c0f5410 2023-11-15T04:10:57.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9a937fe700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 0x7f9a88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:10:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.607+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a84040ff0 con 0x7f9a9c0f5410 2023-11-15T04:10:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.608+0000 7f9a9a7fc700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 0x7f9a88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:10:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.608+0000 7f9a9a7fc700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 0x7f9a88032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9a8c0096a0 tx=0x7f9a8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:10:57.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.613+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a840095f0 con 0x7f9a9c0f5410 2023-11-15T04:10:57.820 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.819+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a9c0fae60 con 0x7f9a9c0f5410 2023-11-15T04:10:57.820 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.820+0000 7f9a937fe700 1 -- 172.21.15.188:0/3027766490 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a8401db70 con 0x7f9a9c0f5410 2023-11-15T04:10:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:10:57.821 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:10:57.823 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 msgr2=0x7f9a88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:57.823 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 0x7f9a88032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9a8c0096a0 tx=0x7f9a8c006b40).stop 2023-11-15T04:10:57.823 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 msgr2=0x7f9a9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:10:57.823 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9a84013510 tx=0x7f9a8400b040).stop 2023-11-15T04:10:57.823 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 shutdown_connections 2023-11-15T04:10:57.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9a8802ff60 0x7f9a88032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:57.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 --2- 172.21.15.188:0/3027766490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c0f5410 0x7f9a9c105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:10:57.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.821+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 >> 172.21.15.188:0/3027766490 conn(0x7f9a9c0f0b70 msgr2=0x7f9a9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:10:57.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.822+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 shutdown_connections 2023-11-15T04:10:57.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:10:57.822+0000 7f9aa18a9700 1 -- 172.21.15.188:0/3027766490 wait complete. 2023-11-15T04:10:57.825 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:10:58.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:58 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:10:57.821184+0000 mon.smithi038 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.188:0/3027766490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:10:59.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:10:59.230 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:10:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:10:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:10:58.065498+0000 mgr.smithi038.bnbogz (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:00.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 -- 172.21.15.188:0/3312182677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 msgr2=0x7fad840f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:00.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 --2- 172.21.15.188:0/3312182677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad840f9150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fad78009fb0 tx=0x7fad78009b30).stop 2023-11-15T04:11:00.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 -- 172.21.15.188:0/3312182677 shutdown_connections 2023-11-15T04:11:00.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 --2- 172.21.15.188:0/3312182677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad840f9150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:00.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 -- 172.21.15.188:0/3312182677 >> 172.21.15.188:0/3312182677 conn(0x7fad840f0b70 msgr2=0x7fad840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:00.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 -- 172.21.15.188:0/3312182677 shutdown_connections 2023-11-15T04:11:00.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 -- 172.21.15.188:0/3312182677 wait complete. 2023-11-15T04:11:00.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.345+0000 7fad89f42700 1 Processor -- start 2023-11-15T04:11:00.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad89f42700 1 -- start start 2023-11-15T04:11:00.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad89f42700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:00.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad89f42700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84106ba0 con 0x7fad84060220 2023-11-15T04:11:00.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad837fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:00.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad837fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57888/0 (socket says 172.21.15.188:57888) 2023-11-15T04:11:00.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad837fe700 1 -- 172.21.15.188:0/2817889445 learned_addr learned my addr 172.21.15.188:0/2817889445 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:00.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad837fe700 1 -- 172.21.15.188:0/2817889445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78005040 con 0x7fad84060220 2023-11-15T04:11:00.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.346+0000 7fad837fe700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fad78013510 tx=0x7fad7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:00.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.347+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78009320 con 0x7fad84060220 2023-11-15T04:11:00.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.347+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad84106da0 con 0x7fad84060220 2023-11-15T04:11:00.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.347+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad78013f00 con 0x7fad84060220 2023-11-15T04:11:00.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.347+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78014dc0 con 0x7fad84060220 2023-11-15T04:11:00.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.347+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad84107230 con 0x7fad84060220 2023-11-15T04:11:00.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad7801d3d0 con 0x7fad84060220 2023-11-15T04:11:00.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84045ad0 con 0x7fad84060220 2023-11-15T04:11:00.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad6bfff700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 0x7fad6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:00.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad78041060 con 0x7fad84060220 2023-11-15T04:11:00.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad82ffd700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 0x7fad6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:00.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.348+0000 7fad82ffd700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 0x7fad6c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fad740096a0 tx=0x7fad74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:00.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.353+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad780095f0 con 0x7fad84060220 2023-11-15T04:11:00.561 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.560+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad841080e0 con 0x7fad84060220 2023-11-15T04:11:00.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.561+0000 7fad6bfff700 1 -- 172.21.15.188:0/2817889445 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad7801cb10 con 0x7fad84060220 2023-11-15T04:11:00.562 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:00.562 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:00.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 msgr2=0x7fad6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:00.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 0x7fad6c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fad740096a0 tx=0x7fad74006b40).stop 2023-11-15T04:11:00.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 msgr2=0x7fad841066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:00.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fad78013510 tx=0x7fad7800b040).stop 2023-11-15T04:11:00.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 shutdown_connections 2023-11-15T04:11:00.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad6c02ff60 0x7fad6c032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:00.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 --2- 172.21.15.188:0/2817889445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad84060220 0x7fad841066b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:00.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 >> 172.21.15.188:0/2817889445 conn(0x7fad840f0b70 msgr2=0x7fad840f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:00.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 shutdown_connections 2023-11-15T04:11:00.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:00.563+0000 7fad89f42700 1 -- 172.21.15.188:0/2817889445 wait complete. 2023-11-15T04:11:00.566 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:00.065860+0000 mgr.smithi038.bnbogz (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:01.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:00.562430+0000 mon.smithi038 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.188:0/2817889445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:01.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:01.965 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:02.066184+0000 mgr.smithi038.bnbogz (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:03.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.208+0000 7f694bcaf700 1 -- 172.21.15.188:0/213623967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 msgr2=0x7f69440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.208+0000 7f694bcaf700 1 --2- 172.21.15.188:0/213623967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f69440f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f6938009fb0 tx=0x7f6938009b30).stop 2023-11-15T04:11:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- 172.21.15.188:0/213623967 shutdown_connections 2023-11-15T04:11:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 --2- 172.21.15.188:0/213623967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f69440f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- 172.21.15.188:0/213623967 >> 172.21.15.188:0/213623967 conn(0x7f69440f0b70 msgr2=0x7f69440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:03.210 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- 172.21.15.188:0/213623967 shutdown_connections 2023-11-15T04:11:03.210 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- 172.21.15.188:0/213623967 wait complete. 2023-11-15T04:11:03.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 Processor -- start 2023-11-15T04:11:03.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- start start 2023-11-15T04:11:03.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.209+0000 7f694bcaf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944060220 con 0x7f69440f5410 2023-11-15T04:11:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f6949a4b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f6949a4b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57916/0 (socket says 172.21.15.188:57916) 2023-11-15T04:11:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f6949a4b700 1 -- 172.21.15.188:0/3940936667 learned_addr learned my addr 172.21.15.188:0/3940936667 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:03.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f6949a4b700 1 -- 172.21.15.188:0/3940936667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938005040 con 0x7f69440f5410 2023-11-15T04:11:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f6949a4b700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f6938013510 tx=0x7f693800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6938009320 con 0x7f69440f5410 2023-11-15T04:11:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6944060420 con 0x7f69440f5410 2023-11-15T04:11:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6938013f00 con 0x7f69440f5410 2023-11-15T04:11:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6938014d90 con 0x7f69440f5410 2023-11-15T04:11:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.210+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6944109b00 con 0x7f69440f5410 2023-11-15T04:11:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.211+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69380095f0 con 0x7f69440f5410 2023-11-15T04:11:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.211+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6944059100 con 0x7f69440f5410 2023-11-15T04:11:03.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.211+0000 7f69367fc700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:03.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.211+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6938040fc0 con 0x7f69440f5410 2023-11-15T04:11:03.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.211+0000 7f694924a700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:03.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.212+0000 7f694924a700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 0x7f6930032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f69400096a0 tx=0x7f6940006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:03.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.216+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6938014420 con 0x7f69440f5410 2023-11-15T04:11:03.423 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.423+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6944045ad0 con 0x7f69440f5410 2023-11-15T04:11:03.424 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.424+0000 7f69367fc700 1 -- 172.21.15.188:0/3940936667 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6938010230 con 0x7f69440f5410 2023-11-15T04:11:03.424 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 msgr2=0x7f6930032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 0x7f6930032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f69400096a0 tx=0x7f6940006b40).stop 2023-11-15T04:11:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 msgr2=0x7f6944109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f6938013510 tx=0x7f693800b040).stop 2023-11-15T04:11:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 shutdown_connections 2023-11-15T04:11:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 --2- 172.21.15.188:0/3940936667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69440f5410 0x7f6944109450 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 >> 172.21.15.188:0/3940936667 conn(0x7f69440f0b70 msgr2=0x7f69440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 shutdown_connections 2023-11-15T04:11:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:03.426+0000 7f694bcaf700 1 -- 172.21.15.188:0/3940936667 wait complete. 2023-11-15T04:11:03.428 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:03.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:03.424523+0000 mon.smithi038 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.188:0/3940936667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:04.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:04.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:04.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:04.066444+0000 mgr.smithi038.bnbogz (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:06.000 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 -- 172.21.15.188:0/290330614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 msgr2=0x7ff0380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:06.001 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 --2- 172.21.15.188:0/290330614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff0380f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ff02c009fb0 tx=0x7ff02c009b30).stop 2023-11-15T04:11:06.001 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 -- 172.21.15.188:0/290330614 shutdown_connections 2023-11-15T04:11:06.001 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 --2- 172.21.15.188:0/290330614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff0380f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:06.001 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 -- 172.21.15.188:0/290330614 >> 172.21.15.188:0/290330614 conn(0x7ff0380f0b70 msgr2=0x7ff0380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 -- 172.21.15.188:0/290330614 shutdown_connections 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:05.999+0000 7ff03e849700 1 -- 172.21.15.188:0/290330614 wait complete. 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.000+0000 7ff03e849700 1 Processor -- start 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.000+0000 7ff03e849700 1 -- start start 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.000+0000 7ff03e849700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:06.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.000+0000 7ff03e849700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038106c20 con 0x7ff0380f5410 2023-11-15T04:11:06.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff037fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:06.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff037fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57936/0 (socket says 172.21.15.188:57936) 2023-11-15T04:11:06.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff037fff700 1 -- 172.21.15.188:0/2913115716 learned_addr learned my addr 172.21.15.188:0/2913115716 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:06.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff037fff700 1 -- 172.21.15.188:0/2913115716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c005040 con 0x7ff0380f5410 2023-11-15T04:11:06.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff037fff700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff02c013510 tx=0x7ff02c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:06.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff02c009320 con 0x7ff0380f5410 2023-11-15T04:11:06.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff038106e20 con 0x7ff0380f5410 2023-11-15T04:11:06.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff02c013f00 con 0x7ff0380f5410 2023-11-15T04:11:06.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff02c014d90 con 0x7ff0380f5410 2023-11-15T04:11:06.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.001+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0381072b0 con 0x7ff0380f5410 2023-11-15T04:11:06.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.002+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff02c014420 con 0x7ff0380f5410 2023-11-15T04:11:06.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.002+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff038045ad0 con 0x7ff0380f5410 2023-11-15T04:11:06.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.002+0000 7ff034ff9700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 0x7ff020032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:06.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.002+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff02c0410e0 con 0x7ff0380f5410 2023-11-15T04:11:06.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.002+0000 7ff0377fe700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 0x7ff020032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:06.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.003+0000 7ff0377fe700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 0x7ff020032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff0280096a0 tx=0x7ff028006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:06.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.007+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff02c0095f0 con 0x7ff0380f5410 2023-11-15T04:11:06.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.213+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0380fbf90 con 0x7ff0380f5410 2023-11-15T04:11:06.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.214+0000 7ff034ff9700 1 -- 172.21.15.188:0/2913115716 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff02c01db70 con 0x7ff0380f5410 2023-11-15T04:11:06.214 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:06.214 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:06.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 msgr2=0x7ff020032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:06.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 0x7ff020032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff0280096a0 tx=0x7ff028006b40).stop 2023-11-15T04:11:06.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 msgr2=0x7ff038106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:06.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff02c013510 tx=0x7ff02c00b040).stop 2023-11-15T04:11:06.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 shutdown_connections 2023-11-15T04:11:06.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff02002ff60 0x7ff020032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:06.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 --2- 172.21.15.188:0/2913115716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0380f5410 0x7ff038106730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:06.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 >> 172.21.15.188:0/2913115716 conn(0x7ff0380f0b70 msgr2=0x7ff0380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:06.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 shutdown_connections 2023-11-15T04:11:06.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:06.215+0000 7ff03e849700 1 -- 172.21.15.188:0/2913115716 wait complete. 2023-11-15T04:11:06.217 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:06.066740+0000 mgr.smithi038.bnbogz (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:06.214080+0000 mon.smithi038 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.188:0/2913115716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:07.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:07.631 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:08.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 -- 172.21.15.188:0/1529440909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 msgr2=0x7fc59c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:08.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 --2- 172.21.15.188:0/1529440909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc58c009fb0 tx=0x7fc58c009b30).stop 2023-11-15T04:11:08.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 -- 172.21.15.188:0/1529440909 shutdown_connections 2023-11-15T04:11:08.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 --2- 172.21.15.188:0/1529440909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:08.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 -- 172.21.15.188:0/1529440909 >> 172.21.15.188:0/1529440909 conn(0x7fc59c0f0b70 msgr2=0x7fc59c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:08.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.826+0000 7fc5a3509700 1 -- 172.21.15.188:0/1529440909 shutdown_connections 2023-11-15T04:11:08.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a3509700 1 -- 172.21.15.188:0/1529440909 wait complete. 2023-11-15T04:11:08.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a3509700 1 Processor -- start 2023-11-15T04:11:08.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a3509700 1 -- start start 2023-11-15T04:11:08.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a3509700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:08.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a3509700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0fb820 con 0x7fc59c0f5410 2023-11-15T04:11:08.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a12a5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:08.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a12a5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54170/0 (socket says 172.21.15.188:54170) 2023-11-15T04:11:08.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.827+0000 7fc5a12a5700 1 -- 172.21.15.188:0/2673048025 learned_addr learned my addr 172.21.15.188:0/2673048025 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:08.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc5a12a5700 1 -- 172.21.15.188:0/2673048025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c005040 con 0x7fc59c0f5410 2023-11-15T04:11:08.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc5a12a5700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc58c009f80 tx=0x7fc58c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:08.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc58c009320 con 0x7fc59c0f5410 2023-11-15T04:11:08.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c0fb9c0 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc58c013ec0 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc58c01ece0 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.828+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c0fbe50 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.829+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc58c01e420 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.829+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c059100 con 0x7fc59c0f5410 2023-11-15T04:11:08.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.829+0000 7fc591ffb700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 0x7fc588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:08.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.829+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc58c040d90 con 0x7fc59c0f5410 2023-11-15T04:11:08.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.829+0000 7fc5a0aa4700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 0x7fc588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:08.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.830+0000 7fc5a0aa4700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 0x7fc588032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc5980096a0 tx=0x7fc598006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:08.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:08.834+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc58c01c920 con 0x7fc59c0f5410 2023-11-15T04:11:09.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.038+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc59c045ad0 con 0x7fc59c0f5410 2023-11-15T04:11:09.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.038+0000 7fc591ffb700 1 -- 172.21.15.188:0/2673048025 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc58c02c2d0 con 0x7fc59c0f5410 2023-11-15T04:11:09.039 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:09.039 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:09.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 msgr2=0x7fc588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:09.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 0x7fc588032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc5980096a0 tx=0x7fc598006b40).stop 2023-11-15T04:11:09.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 msgr2=0x7fc59c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:09.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc58c009f80 tx=0x7fc58c00b040).stop 2023-11-15T04:11:09.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 shutdown_connections 2023-11-15T04:11:09.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc58802ff60 0x7fc588032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:09.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 --2- 172.21.15.188:0/2673048025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc59c0f5410 0x7fc59c0fd3a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:09.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 >> 172.21.15.188:0/2673048025 conn(0x7fc59c0f0b70 msgr2=0x7fc59c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:09.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 shutdown_connections 2023-11-15T04:11:09.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:09.040+0000 7fc5a3509700 1 -- 172.21.15.188:0/2673048025 wait complete. 2023-11-15T04:11:09.042 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:08.067020+0000 mgr.smithi038.bnbogz (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:09.038948+0000 mon.smithi038 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.188:0/2673048025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:10.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:10.448 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:10.067318+0000 mgr.smithi038.bnbogz (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:11.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- 172.21.15.188:0/14042549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 msgr2=0x7f0cbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:11.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/14042549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc0f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0cac009fb0 tx=0x7f0cac009b30).stop 2023-11-15T04:11:11.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- 172.21.15.188:0/14042549 shutdown_connections 2023-11-15T04:11:11.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/14042549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc0f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:11.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- 172.21.15.188:0/14042549 >> 172.21.15.188:0/14042549 conn(0x7f0cbc0f0b70 msgr2=0x7f0cbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:11.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- 172.21.15.188:0/14042549 shutdown_connections 2023-11-15T04:11:11.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- 172.21.15.188:0/14042549 wait complete. 2023-11-15T04:11:11.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 Processor -- start 2023-11-15T04:11:11.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.626+0000 7f0cc3dad700 1 -- start start 2023-11-15T04:11:11.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc3dad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:11.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc3dad700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc060240 con 0x7f0cbc0f5410 2023-11-15T04:11:11.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc1b49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:11.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc1b49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54184/0 (socket says 172.21.15.188:54184) 2023-11-15T04:11:11.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc1b49700 1 -- 172.21.15.188:0/441375155 learned_addr learned my addr 172.21.15.188:0/441375155 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:11.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc1b49700 1 -- 172.21.15.188:0/441375155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac005040 con 0x7f0cbc0f5410 2023-11-15T04:11:11.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc1b49700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f0cac014000 tx=0x7f0cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:11.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac0095f0 con 0x7f0cbc0f5410 2023-11-15T04:11:11.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.627+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc060440 con 0x7f0cbc0f5410 2023-11-15T04:11:11.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cac007e70 con 0x7f0cbc0f5410 2023-11-15T04:11:11.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac014ea0 con 0x7f0cbc0f5410 2023-11-15T04:11:11.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc109ae0 con 0x7f0cbc0f5410 2023-11-15T04:11:11.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cac025070 con 0x7f0cbc0f5410 2023-11-15T04:11:11.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc045ad0 con 0x7f0cbc0f5410 2023-11-15T04:11:11.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cb27fc700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 0x7f0ca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:11.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.628+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cac040940 con 0x7f0cbc0f5410 2023-11-15T04:11:11.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.629+0000 7f0cc1348700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 0x7f0ca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:11.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.629+0000 7f0cc1348700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 0x7f0ca8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0cb80096a0 tx=0x7f0cb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:11.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.633+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cac0079c0 con 0x7f0cbc0f5410 2023-11-15T04:11:11.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.841+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cbc0fae60 con 0x7f0cbc0f5410 2023-11-15T04:11:11.841 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.841+0000 7f0cb27fc700 1 -- 172.21.15.188:0/441375155 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cac00f300 con 0x7f0cbc0f5410 2023-11-15T04:11:11.841 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:11.842 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:11.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 msgr2=0x7f0ca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 0x7f0ca8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0cb80096a0 tx=0x7f0cb8006b40).stop 2023-11-15T04:11:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 msgr2=0x7f0cbc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f0cac014000 tx=0x7f0cac00b040).stop 2023-11-15T04:11:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 shutdown_connections 2023-11-15T04:11:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ca802ff60 0x7f0ca8032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:11.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 --2- 172.21.15.188:0/441375155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cbc0f5410 0x7f0cbc109430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:11.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 >> 172.21.15.188:0/441375155 conn(0x7f0cbc0f0b70 msgr2=0x7f0cbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:11.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 shutdown_connections 2023-11-15T04:11:11.844 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:11.843+0000 7f0cc3dad700 1 -- 172.21.15.188:0/441375155 wait complete. 2023-11-15T04:11:11.845 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:11.841606+0000 mon.smithi038 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.188:0/441375155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:12.067660+0000 mgr.smithi038.bnbogz (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:13.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:13.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:14.445 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 -- 172.21.15.188:0/142526109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 msgr2=0x7fb35c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:14.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 --2- 172.21.15.188:0/142526109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c0f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb358009fb0 tx=0x7fb358009b30).stop 2023-11-15T04:11:14.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 -- 172.21.15.188:0/142526109 shutdown_connections 2023-11-15T04:11:14.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 --2- 172.21.15.188:0/142526109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c0f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:14.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 -- 172.21.15.188:0/142526109 >> 172.21.15.188:0/142526109 conn(0x7fb35c0f0b70 msgr2=0x7fb35c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:14.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 -- 172.21.15.188:0/142526109 shutdown_connections 2023-11-15T04:11:14.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.443+0000 7fb363aef700 1 -- 172.21.15.188:0/142526109 wait complete. 2023-11-15T04:11:14.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.444+0000 7fb363aef700 1 Processor -- start 2023-11-15T04:11:14.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.444+0000 7fb363aef700 1 -- start start 2023-11-15T04:11:14.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.444+0000 7fb363aef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:14.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.444+0000 7fb363aef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c060240 con 0x7fb35c0f5410 2023-11-15T04:11:14.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb36188b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:14.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb36188b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54208/0 (socket says 172.21.15.188:54208) 2023-11-15T04:11:14.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb36188b700 1 -- 172.21.15.188:0/2101350319 learned_addr learned my addr 172.21.15.188:0/2101350319 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:14.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb36188b700 1 -- 172.21.15.188:0/2101350319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358005040 con 0x7fb35c0f5410 2023-11-15T04:11:14.449 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb36188b700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb358014000 tx=0x7fb35800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:14.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3580095f0 con 0x7fb35c0f5410 2023-11-15T04:11:14.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c060440 con 0x7fb35c0f5410 2023-11-15T04:11:14.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb358007e70 con 0x7fb35c0f5410 2023-11-15T04:11:14.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358014ea0 con 0x7fb35c0f5410 2023-11-15T04:11:14.451 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.445+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c109ae0 con 0x7fb35c0f5410 2023-11-15T04:11:14.451 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.446+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c045ad0 con 0x7fb35c0f5410 2023-11-15T04:11:14.451 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.446+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb358025070 con 0x7fb35c0f5410 2023-11-15T04:11:14.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.446+0000 7fb3527fc700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:14.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.446+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3580409f0 con 0x7fb35c0f5410 2023-11-15T04:11:14.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.446+0000 7fb36108a700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:14.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.447+0000 7fb36108a700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 0x7fb348032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb34c0096a0 tx=0x7fb34c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:14.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.451+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3580079c0 con 0x7fb35c0f5410 2023-11-15T04:11:14.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.653+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb35c0fae60 con 0x7fb35c0f5410 2023-11-15T04:11:14.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.654+0000 7fb3527fc700 1 -- 172.21.15.188:0/2101350319 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb35800f300 con 0x7fb35c0f5410 2023-11-15T04:11:14.655 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:14.655 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:14.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 msgr2=0x7fb348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:14.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 0x7fb348032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb34c0096a0 tx=0x7fb34c006b40).stop 2023-11-15T04:11:14.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 msgr2=0x7fb35c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:14.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb358014000 tx=0x7fb35800b040).stop 2023-11-15T04:11:14.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 shutdown_connections 2023-11-15T04:11:14.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb34802ff60 0x7fb348032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:14.657 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 --2- 172.21.15.188:0/2101350319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb35c0f5410 0x7fb35c109430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:14.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 >> 172.21.15.188:0/2101350319 conn(0x7fb35c0f0b70 msgr2=0x7fb35c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:14.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 shutdown_connections 2023-11-15T04:11:14.658 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:14.655+0000 7fb363aef700 1 -- 172.21.15.188:0/2101350319 wait complete. 2023-11-15T04:11:14.658 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:14.068044+0000 mgr.smithi038.bnbogz (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:14.655244+0000 mon.smithi038 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.188:0/2101350319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:16.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:16.050 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:17.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f1770f8e700 1 -- 172.21.15.188:0/3975152972 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f175c0138a0 con 0x7f176c0f6510 2023-11-15T04:11:17.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f17741f4700 1 -- 172.21.15.188:0/3975152972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 msgr2=0x7f176c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:17.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f17741f4700 1 --2- 172.21.15.188:0/3975152972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c0f6930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f175c009fb0 tx=0x7f175c009b30).stop 2023-11-15T04:11:17.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f17741f4700 1 -- 172.21.15.188:0/3975152972 shutdown_connections 2023-11-15T04:11:17.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f17741f4700 1 --2- 172.21.15.188:0/3975152972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c0f6930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:17.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.248+0000 7f17741f4700 1 -- 172.21.15.188:0/3975152972 >> 172.21.15.188:0/3975152972 conn(0x7f176c0f1c70 msgr2=0x7f176c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:17.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 -- 172.21.15.188:0/3975152972 shutdown_connections 2023-11-15T04:11:17.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 -- 172.21.15.188:0/3975152972 wait complete. 2023-11-15T04:11:17.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 Processor -- start 2023-11-15T04:11:17.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 -- start start 2023-11-15T04:11:17.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:17.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.249+0000 7f17741f4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c108df0 con 0x7f176c0f6510 2023-11-15T04:11:17.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f1771f90700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:17.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f1771f90700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54232/0 (socket says 172.21.15.188:54232) 2023-11-15T04:11:17.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f1771f90700 1 -- 172.21.15.188:0/3893691538 learned_addr learned my addr 172.21.15.188:0/3893691538 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:17.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f1771f90700 1 -- 172.21.15.188:0/3893691538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c005040 con 0x7f176c0f6510 2023-11-15T04:11:17.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f1771f90700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f175c013510 tx=0x7f175c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:17.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f175c009320 con 0x7f176c0f6510 2023-11-15T04:11:17.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c108ff0 con 0x7f176c0f6510 2023-11-15T04:11:17.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f175c013f00 con 0x7f176c0f6510 2023-11-15T04:11:17.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f175c014d90 con 0x7f176c0f6510 2023-11-15T04:11:17.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.250+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c109480 con 0x7f176c0f6510 2023-11-15T04:11:17.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.251+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f175c014420 con 0x7f176c0f6510 2023-11-15T04:11:17.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.251+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c045ad0 con 0x7f176c0f6510 2023-11-15T04:11:17.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.251+0000 7f17627fc700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 0x7f1758032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:17.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.251+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f175c041070 con 0x7f176c0f6510 2023-11-15T04:11:17.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.252+0000 7f177178f700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 0x7f1758032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:17.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.252+0000 7f177178f700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 0x7f1758032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f17680096a0 tx=0x7f1768006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:17.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.256+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f175c0095f0 con 0x7f176c0f6510 2023-11-15T04:11:17.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.459+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f176c06aec0 con 0x7f176c0f6510 2023-11-15T04:11:17.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.460+0000 7f17627fc700 1 -- 172.21.15.188:0/3893691538 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f175c01db70 con 0x7f176c0f6510 2023-11-15T04:11:17.460 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:17.461 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:17.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 msgr2=0x7f1758032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:17.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 0x7f1758032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f17680096a0 tx=0x7f1768006b40).stop 2023-11-15T04:11:17.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 msgr2=0x7f176c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:17.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f175c013510 tx=0x7f175c00b040).stop 2023-11-15T04:11:17.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 shutdown_connections 2023-11-15T04:11:17.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f175802ff60 0x7f1758032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:17.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 --2- 172.21.15.188:0/3893691538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c0f6510 0x7f176c108900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:17.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 >> 172.21.15.188:0/3893691538 conn(0x7f176c0f1c70 msgr2=0x7f176c0f2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:17.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 shutdown_connections 2023-11-15T04:11:17.464 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:17.461+0000 7f17741f4700 1 -- 172.21.15.188:0/3893691538 wait complete. 2023-11-15T04:11:17.464 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:16.068379+0000 mgr.smithi038.bnbogz (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:18.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:18 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:17.460982+0000 mon.smithi038 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.188:0/3893691538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:18.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:18.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:18.068649+0000 mgr.smithi038.bnbogz (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:20.134 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 -- 172.21.15.188:0/4263473150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 msgr2=0x7f3a600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 --2- 172.21.15.188:0/4263473150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a600f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3a58009960 tx=0x7f3a58004d10).stop 2023-11-15T04:11:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 -- 172.21.15.188:0/4263473150 shutdown_connections 2023-11-15T04:11:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 --2- 172.21.15.188:0/4263473150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a600f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 -- 172.21.15.188:0/4263473150 >> 172.21.15.188:0/4263473150 conn(0x7f3a600f1c70 msgr2=0x7f3a600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 -- 172.21.15.188:0/4263473150 shutdown_connections 2023-11-15T04:11:20.136 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.131+0000 7f3a66f21700 1 -- 172.21.15.188:0/4263473150 wait complete. 2023-11-15T04:11:20.136 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a66f21700 1 Processor -- start 2023-11-15T04:11:20.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a66f21700 1 -- start start 2023-11-15T04:11:20.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a66f21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:20.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a66f21700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60106c20 con 0x7f3a600f6510 2023-11-15T04:11:20.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a64cbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:20.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a64cbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49172/0 (socket says 172.21.15.188:49172) 2023-11-15T04:11:20.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.132+0000 7f3a64cbd700 1 -- 172.21.15.188:0/661762654 learned_addr learned my addr 172.21.15.188:0/661762654 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:20.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.133+0000 7f3a64cbd700 1 -- 172.21.15.188:0/661762654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a58009640 con 0x7f3a600f6510 2023-11-15T04:11:20.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.133+0000 7f3a64cbd700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3a58004b70 tx=0x7f3a58004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:20.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.134+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a58018070 con 0x7f3a600f6510 2023-11-15T04:11:20.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.134+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a5800b870 con 0x7f3a600f6510 2023-11-15T04:11:20.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.134+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a60106e20 con 0x7f3a600f6510 2023-11-15T04:11:20.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.134+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a5801c560 con 0x7f3a600f6510 2023-11-15T04:11:20.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.134+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a601072b0 con 0x7f3a600f6510 2023-11-15T04:11:20.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.135+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a58013630 con 0x7f3a600f6510 2023-11-15T04:11:20.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.135+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a60045ad0 con 0x7f3a600f6510 2023-11-15T04:11:20.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.135+0000 7f3a5d7fa700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 0x7f3a48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:20.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.135+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a58041430 con 0x7f3a600f6510 2023-11-15T04:11:20.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.136+0000 7f3a5ffff700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 0x7f3a48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:20.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.136+0000 7f3a5ffff700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 0x7f3a48032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3a540096a0 tx=0x7f3a54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:20.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.140+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a58004810 con 0x7f3a600f6510 2023-11-15T04:11:20.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.343+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a600fbf60 con 0x7f3a600f6510 2023-11-15T04:11:20.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.344+0000 7f3a5d7fa700 1 -- 172.21.15.188:0/661762654 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a58011b10 con 0x7f3a600f6510 2023-11-15T04:11:20.347 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:20.347 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:20.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 msgr2=0x7f3a48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:20.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 0x7f3a48032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3a540096a0 tx=0x7f3a54006b40).stop 2023-11-15T04:11:20.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 msgr2=0x7f3a60106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:20.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3a58004b70 tx=0x7f3a58004d10).stop 2023-11-15T04:11:20.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 shutdown_connections 2023-11-15T04:11:20.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a4802ffb0 0x7f3a48032470 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:20.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 --2- 172.21.15.188:0/661762654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a600f6510 0x7f3a60106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:20.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.345+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 >> 172.21.15.188:0/661762654 conn(0x7f3a600f1c70 msgr2=0x7f3a600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:20.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.346+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 shutdown_connections 2023-11-15T04:11:20.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:20.346+0000 7f3a66f21700 1 -- 172.21.15.188:0/661762654 wait complete. 2023-11-15T04:11:20.351 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:20.068981+0000 mgr.smithi038.bnbogz (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:20.345334+0000 mon.smithi038 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.188:0/661762654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:21.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:21.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:22.069288+0000 mgr.smithi038.bnbogz (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:22.972 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.970+0000 7f6960bab700 1 -- 172.21.15.188:0/2033849920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 msgr2=0x7f695c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:22.972 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.970+0000 7f6960bab700 1 --2- 172.21.15.188:0/2033849920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6944009fb0 tx=0x7f6944009b30).stop 2023-11-15T04:11:22.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- 172.21.15.188:0/2033849920 shutdown_connections 2023-11-15T04:11:22.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 --2- 172.21.15.188:0/2033849920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:22.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- 172.21.15.188:0/2033849920 >> 172.21.15.188:0/2033849920 conn(0x7f695c0f0b70 msgr2=0x7f695c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:22.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- 172.21.15.188:0/2033849920 shutdown_connections 2023-11-15T04:11:22.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- 172.21.15.188:0/2033849920 wait complete. 2023-11-15T04:11:22.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 Processor -- start 2023-11-15T04:11:22.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- start start 2023-11-15T04:11:22.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:22.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.971+0000 7f6960bab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c107240 con 0x7f695c0f5410 2023-11-15T04:11:22.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f695a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:22.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f695a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49192/0 (socket says 172.21.15.188:49192) 2023-11-15T04:11:22.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f695a59c700 1 -- 172.21.15.188:0/3787866355 learned_addr learned my addr 172.21.15.188:0/3787866355 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:22.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f695a59c700 1 -- 172.21.15.188:0/3787866355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944005040 con 0x7f695c0f5410 2023-11-15T04:11:22.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f695a59c700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6944013510 tx=0x7f694400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:22.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6944009320 con 0x7f695c0f5410 2023-11-15T04:11:22.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c060220 con 0x7f695c0f5410 2023-11-15T04:11:22.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c10bd40 con 0x7f695c0f5410 2023-11-15T04:11:22.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6944013f00 con 0x7f695c0f5410 2023-11-15T04:11:22.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.972+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6944014d60 con 0x7f695c0f5410 2023-11-15T04:11:22.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.973+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6944014420 con 0x7f695c0f5410 2023-11-15T04:11:22.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.973+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6940004fa0 con 0x7f695c0f5410 2023-11-15T04:11:22.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.973+0000 7f6952ffd700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 0x7f6948032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:22.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.973+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6944041240 con 0x7f695c0f5410 2023-11-15T04:11:22.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.974+0000 7f6959d9b700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 0x7f6948032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:22.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.974+0000 7f6959d9b700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 0x7f6948032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f694c0096a0 tx=0x7f694c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:22.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:22.978+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6944012b20 con 0x7f695c0f5410 2023-11-15T04:11:23.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.185+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6940005670 con 0x7f695c0f5410 2023-11-15T04:11:23.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.186+0000 7f6952ffd700 1 -- 172.21.15.188:0/3787866355 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6944012b20 con 0x7f695c0f5410 2023-11-15T04:11:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:23.187 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:23.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 msgr2=0x7f6948032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:23.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 0x7f6948032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f694c0096a0 tx=0x7f694c006b40).stop 2023-11-15T04:11:23.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 msgr2=0x7f695c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:23.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6944013510 tx=0x7f694400b040).stop 2023-11-15T04:11:23.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 shutdown_connections 2023-11-15T04:11:23.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f694802ff60 0x7f6948032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:23.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 --2- 172.21.15.188:0/3787866355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f695c0f5410 0x7f695c10b850 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:23.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 >> 172.21.15.188:0/3787866355 conn(0x7f695c0f0b70 msgr2=0x7f695c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:23.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 shutdown_connections 2023-11-15T04:11:23.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:23.187+0000 7f6960bab700 1 -- 172.21.15.188:0/3787866355 wait complete. 2023-11-15T04:11:23.194 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:23.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:23.187008+0000 mon.smithi038 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.188:0/3787866355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:24.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:24.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:24.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:24 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:24.069622+0000 mgr.smithi038.bnbogz (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:25.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 -- 172.21.15.188:0/3475412329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 msgr2=0x7f579c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:25.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 --2- 172.21.15.188:0/3475412329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c0f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5790009fb0 tx=0x7f5790009b30).stop 2023-11-15T04:11:25.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 -- 172.21.15.188:0/3475412329 shutdown_connections 2023-11-15T04:11:25.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 --2- 172.21.15.188:0/3475412329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c0f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:25.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 -- 172.21.15.188:0/3475412329 >> 172.21.15.188:0/3475412329 conn(0x7f579c0f0b70 msgr2=0x7f579c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:25.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 -- 172.21.15.188:0/3475412329 shutdown_connections 2023-11-15T04:11:25.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.801+0000 7f57a18dc700 1 -- 172.21.15.188:0/3475412329 wait complete. 2023-11-15T04:11:25.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.802+0000 7f57a18dc700 1 Processor -- start 2023-11-15T04:11:25.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.802+0000 7f57a18dc700 1 -- start start 2023-11-15T04:11:25.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.802+0000 7f57a18dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:25.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.802+0000 7f57a18dc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c060240 con 0x7f579c0f5410 2023-11-15T04:11:25.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.802+0000 7f579affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:25.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f579affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49212/0 (socket says 172.21.15.188:49212) 2023-11-15T04:11:25.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f579affd700 1 -- 172.21.15.188:0/650638760 learned_addr learned my addr 172.21.15.188:0/650638760 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:25.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f579affd700 1 -- 172.21.15.188:0/650638760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790005040 con 0x7f579c0f5410 2023-11-15T04:11:25.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f579affd700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5790013510 tx=0x7f579000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:25.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5790009320 con 0x7f579c0f5410 2023-11-15T04:11:25.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c060440 con 0x7f579c0f5410 2023-11-15T04:11:25.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5790013f00 con 0x7f579c0f5410 2023-11-15T04:11:25.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5790014d90 con 0x7f579c0f5410 2023-11-15T04:11:25.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.803+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c109ae0 con 0x7f579c0f5410 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.804+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5790014420 con 0x7f579c0f5410 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.804+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579c045ad0 con 0x7f579c0f5410 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.804+0000 7f57837fe700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.804+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5790040fb0 con 0x7f579c0f5410 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.804+0000 7f579a7fc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:25.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.805+0000 7f579a7fc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:25.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:25.809+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57900095f0 con 0x7f579c0f5410 2023-11-15T04:11:26.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.017+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f579c0fae60 con 0x7f579c0f5410 2023-11-15T04:11:26.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.018+0000 7f57837fe700 1 -- 172.21.15.188:0/650638760 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f579001db70 con 0x7f579c0f5410 2023-11-15T04:11:26.019 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:26.020 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:26.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 msgr2=0x7f5784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:26.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).stop 2023-11-15T04:11:26.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 msgr2=0x7f579c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:26.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5790013510 tx=0x7f579000b040).stop 2023-11-15T04:11:26.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 shutdown_connections 2023-11-15T04:11:26.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:26.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 --2- 172.21.15.188:0/650638760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f579c0f5410 0x7f579c109430 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:26.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 >> 172.21.15.188:0/650638760 conn(0x7f579c0f0b70 msgr2=0x7f579c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:26.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 shutdown_connections 2023-11-15T04:11:26.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:26.020+0000 7f57a18dc700 1 -- 172.21.15.188:0/650638760 wait complete. 2023-11-15T04:11:26.023 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:26.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:26.019480+0000 mon.smithi038 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.188:0/650638760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:27.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:27.437 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:27.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:26.069925+0000 mgr.smithi038.bnbogz (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:28.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 -- 172.21.15.188:0/3793646102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 msgr2=0x7f40f40f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:28.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/3793646102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f40f6850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f40e4009fb0 tx=0x7f40e4009b30).stop 2023-11-15T04:11:28.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 -- 172.21.15.188:0/3793646102 shutdown_connections 2023-11-15T04:11:28.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/3793646102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f40f6850 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:28.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 -- 172.21.15.188:0/3793646102 >> 172.21.15.188:0/3793646102 conn(0x7f40f40f1b90 msgr2=0x7f40f40f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:28.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.612+0000 7f40f9ed5700 1 -- 172.21.15.188:0/3793646102 shutdown_connections 2023-11-15T04:11:28.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.613+0000 7f40f9ed5700 1 -- 172.21.15.188:0/3793646102 wait complete. 2023-11-15T04:11:28.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.613+0000 7f40f9ed5700 1 Processor -- start 2023-11-15T04:11:28.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.613+0000 7f40f9ed5700 1 -- start start 2023-11-15T04:11:28.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.613+0000 7f40f9ed5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:28.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.613+0000 7f40f9ed5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4106ec0 con 0x7f40f40f6430 2023-11-15T04:11:28.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:28.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59362/0 (socket says 172.21.15.188:59362) 2023-11-15T04:11:28.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f37fe700 1 -- 172.21.15.188:0/1898339772 learned_addr learned my addr 172.21.15.188:0/1898339772 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:28.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f37fe700 1 -- 172.21.15.188:0/1898339772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4005040 con 0x7f40f40f6430 2023-11-15T04:11:28.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f37fe700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f40e4013510 tx=0x7f40e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:28.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e4009320 con 0x7f40f40f6430 2023-11-15T04:11:28.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f41070c0 con 0x7f40f40f6430 2023-11-15T04:11:28.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40e4013f00 con 0x7f40f40f6430 2023-11-15T04:11:28.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40e4014d90 con 0x7f40f40f6430 2023-11-15T04:11:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.614+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f4107550 con 0x7f40f40f6430 2023-11-15T04:11:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.615+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40e40095f0 con 0x7f40f40f6430 2023-11-15T04:11:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.615+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f4045ad0 con 0x7f40f40f6430 2023-11-15T04:11:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.615+0000 7f40dbfff700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:28.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.615+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40e4040fc0 con 0x7f40f40f6430 2023-11-15T04:11:28.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.616+0000 7f40f2ffd700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:28.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.616+0000 7f40f2ffd700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f40e80096a0 tx=0x7f40e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:28.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.620+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40e4014420 con 0x7f40f40f6430 2023-11-15T04:11:28.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.827+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40f405ea50 con 0x7f40f40f6430 2023-11-15T04:11:28.829 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.828+0000 7f40dbfff700 1 -- 172.21.15.188:0/1898339772 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40e4010230 con 0x7f40f40f6430 2023-11-15T04:11:28.829 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:28.830 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:28.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 msgr2=0x7f40dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:28.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f40e80096a0 tx=0x7f40e8006b40).stop 2023-11-15T04:11:28.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 msgr2=0x7f40f41069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:28.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f40e4013510 tx=0x7f40e400b040).stop 2023-11-15T04:11:28.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 shutdown_connections 2023-11-15T04:11:28.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:28.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 --2- 172.21.15.188:0/1898339772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40f40f6430 0x7f40f41069d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:28.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 >> 172.21.15.188:0/1898339772 conn(0x7f40f40f1b90 msgr2=0x7f40f40f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:28.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 shutdown_connections 2023-11-15T04:11:28.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:28.830+0000 7f40f9ed5700 1 -- 172.21.15.188:0/1898339772 wait complete. 2023-11-15T04:11:28.835 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:28.070148+0000 mgr.smithi038.bnbogz (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:28.829443+0000 mon.smithi038 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.188:0/1898339772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:30.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:30.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:31.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/2897586507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 msgr2=0x7f5fb80f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:31.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/2897586507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb80f9170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5fa8009fb0 tx=0x7f5fa8009b30).stop 2023-11-15T04:11:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/2897586507 shutdown_connections 2023-11-15T04:11:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/2897586507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb80f9170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/2897586507 >> 172.21.15.188:0/2897586507 conn(0x7f5fb80f0b70 msgr2=0x7f5fb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/2897586507 shutdown_connections 2023-11-15T04:11:31.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.477+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/2897586507 wait complete. 2023-11-15T04:11:31.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbf9c7700 1 Processor -- start 2023-11-15T04:11:31.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbf9c7700 1 -- start start 2023-11-15T04:11:31.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbf9c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:31.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbf9c7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb8106bb0 con 0x7f5fb8060200 2023-11-15T04:11:31.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbd763700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:31.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbd763700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59382/0 (socket says 172.21.15.188:59382) 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.478+0000 7f5fbd763700 1 -- 172.21.15.188:0/1403729921 learned_addr learned my addr 172.21.15.188:0/1403729921 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fbd763700 1 -- 172.21.15.188:0/1403729921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8005040 con 0x7f5fb8060200 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fbd763700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5fa8013510 tx=0x7f5fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa8009320 con 0x7f5fb8060200 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb8106db0 con 0x7f5fb8060200 2023-11-15T04:11:31.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fa8013f00 con 0x7f5fb8060200 2023-11-15T04:11:31.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa8014dc0 con 0x7f5fb8060200 2023-11-15T04:11:31.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.479+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb8107240 con 0x7f5fb8060200 2023-11-15T04:11:31.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.480+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fa80095f0 con 0x7f5fb8060200 2023-11-15T04:11:31.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.480+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb8045ad0 con 0x7f5fb8060200 2023-11-15T04:11:31.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.480+0000 7f5fadffb700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 0x7f5fa4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:31.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.480+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fa8040ff0 con 0x7f5fb8060200 2023-11-15T04:11:31.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.480+0000 7f5fbcf62700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 0x7f5fa4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:31.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.481+0000 7f5fbcf62700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 0x7f5fa4032470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fb40096a0 tx=0x7f5fb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:31.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.485+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fa806d2b0 con 0x7f5fb8060200 2023-11-15T04:11:31.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:30.070409+0000 mgr.smithi038.bnbogz (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:31.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.686+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fb81080f0 con 0x7f5fb8060200 2023-11-15T04:11:31.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.687+0000 7f5fadffb700 1 -- 172.21.15.188:0/1403729921 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fa8014420 con 0x7f5fb8060200 2023-11-15T04:11:31.688 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:31.688 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:31.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 msgr2=0x7f5fa4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:31.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 0x7f5fa4032470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fb40096a0 tx=0x7f5fb4006b40).stop 2023-11-15T04:11:31.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 msgr2=0x7f5fb81066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:31.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5fa8013510 tx=0x7f5fa800b040).stop 2023-11-15T04:11:31.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 shutdown_connections 2023-11-15T04:11:31.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa402ffb0 0x7f5fa4032470 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:31.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 --2- 172.21.15.188:0/1403729921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb8060200 0x7f5fb81066c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:31.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 >> 172.21.15.188:0/1403729921 conn(0x7f5fb80f0b70 msgr2=0x7f5fb80f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:31.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 shutdown_connections 2023-11-15T04:11:31.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:31.689+0000 7f5fbf9c7700 1 -- 172.21.15.188:0/1403729921 wait complete. 2023-11-15T04:11:31.692 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:31.688555+0000 mon.smithi038 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.188:0/1403729921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:32.070730+0000 mgr.smithi038.bnbogz (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:33.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:33.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:34.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- 172.21.15.188:0/1199440719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 msgr2=0x7fc5a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:34.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/1199440719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fc59c009fb0 tx=0x7fc59c009b30).stop 2023-11-15T04:11:34.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- 172.21.15.188:0/1199440719 shutdown_connections 2023-11-15T04:11:34.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/1199440719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:34.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- 172.21.15.188:0/1199440719 >> 172.21.15.188:0/1199440719 conn(0x7fc5a80f0b70 msgr2=0x7fc5a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:34.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- 172.21.15.188:0/1199440719 shutdown_connections 2023-11-15T04:11:34.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- 172.21.15.188:0/1199440719 wait complete. 2023-11-15T04:11:34.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 Processor -- start 2023-11-15T04:11:34.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.258+0000 7fc5af0b6700 1 -- start start 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5af0b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5af0b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80fb820 con 0x7fc5a80f5410 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5ace52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5ace52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59412/0 (socket says 172.21.15.188:59412) 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5ace52700 1 -- 172.21.15.188:0/286451592 learned_addr learned my addr 172.21.15.188:0/286451592 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:34.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.259+0000 7fc5ace52700 1 -- 172.21.15.188:0/286451592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c005040 con 0x7fc5a80f5410 2023-11-15T04:11:34.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5ace52700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc59c013510 tx=0x7fc59c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:34.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59c009320 con 0x7fc5a80f5410 2023-11-15T04:11:34.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a80fb9c0 con 0x7fc5a80f5410 2023-11-15T04:11:34.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc59c013f00 con 0x7fc5a80f5410 2023-11-15T04:11:34.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59c014d90 con 0x7fc5a80f5410 2023-11-15T04:11:34.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.260+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a80fbe50 con 0x7fc5a80f5410 2023-11-15T04:11:34.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc59c014420 con 0x7fc5a80f5410 2023-11-15T04:11:34.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a8059100 con 0x7fc5a80f5410 2023-11-15T04:11:34.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5a57fa700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 0x7fc598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:34.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc59c041000 con 0x7fc5a80f5410 2023-11-15T04:11:34.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5a7fff700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 0x7fc598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:34.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.261+0000 7fc5a7fff700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 0x7fc598032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fc5940096a0 tx=0x7fc594006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:34.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.266+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc59c0095f0 con 0x7fc5a80f5410 2023-11-15T04:11:34.474 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.474+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5a80fae60 con 0x7fc5a80f5410 2023-11-15T04:11:34.475 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.474+0000 7fc5a57fa700 1 -- 172.21.15.188:0/286451592 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc59c01db70 con 0x7fc5a80f5410 2023-11-15T04:11:34.475 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:34.476 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:34.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 msgr2=0x7fc598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:34.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 0x7fc598032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fc5940096a0 tx=0x7fc594006b40).stop 2023-11-15T04:11:34.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 msgr2=0x7fc5a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:34.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc59c013510 tx=0x7fc59c00b040).stop 2023-11-15T04:11:34.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 shutdown_connections 2023-11-15T04:11:34.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc59802ff60 0x7fc598032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:34.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 --2- 172.21.15.188:0/286451592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:34.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 >> 172.21.15.188:0/286451592 conn(0x7fc5a80f0b70 msgr2=0x7fc5a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:34.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 shutdown_connections 2023-11-15T04:11:34.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:34.476+0000 7fc5af0b6700 1 -- 172.21.15.188:0/286451592 wait complete. 2023-11-15T04:11:34.479 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:35.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:34.071023+0000 mgr.smithi038.bnbogz (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:35.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:34.475770+0000 mon.smithi038 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.188:0/286451592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:35.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:35.862 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:37.061 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 -- 172.21.15.188:0/1657973912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 msgr2=0x7f47dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:37.061 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/1657973912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc0f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f47cc009fb0 tx=0x7f47cc009b30).stop 2023-11-15T04:11:37.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 -- 172.21.15.188:0/1657973912 shutdown_connections 2023-11-15T04:11:37.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/1657973912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc0f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:37.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 -- 172.21.15.188:0/1657973912 >> 172.21.15.188:0/1657973912 conn(0x7f47dc0f1c70 msgr2=0x7f47dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:37.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 -- 172.21.15.188:0/1657973912 shutdown_connections 2023-11-15T04:11:37.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.060+0000 7f47e3a2b700 1 -- 172.21.15.188:0/1657973912 wait complete. 2023-11-15T04:11:37.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e3a2b700 1 Processor -- start 2023-11-15T04:11:37.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e3a2b700 1 -- start start 2023-11-15T04:11:37.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e3a2b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:37.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e3a2b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc106c20 con 0x7f47dc0f6510 2023-11-15T04:11:37.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e17c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:37.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e17c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59432/0 (socket says 172.21.15.188:59432) 2023-11-15T04:11:37.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e17c7700 1 -- 172.21.15.188:0/3652662883 learned_addr learned my addr 172.21.15.188:0/3652662883 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:37.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.061+0000 7f47e17c7700 1 -- 172.21.15.188:0/3652662883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc005040 con 0x7f47dc0f6510 2023-11-15T04:11:37.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47e17c7700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f47cc013510 tx=0x7f47cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:37.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc009320 con 0x7f47dc0f6510 2023-11-15T04:11:37.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc106e20 con 0x7f47dc0f6510 2023-11-15T04:11:37.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47cc013f00 con 0x7f47dc0f6510 2023-11-15T04:11:37.067 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc014d90 con 0x7f47dc0f6510 2023-11-15T04:11:37.067 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.062+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc1072b0 con 0x7f47dc0f6510 2023-11-15T04:11:37.067 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.063+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47cc0263a0 con 0x7f47dc0f6510 2023-11-15T04:11:37.067 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.063+0000 7f47d1ffb700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:37.067 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.063+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47cc040fe0 con 0x7f47dc0f6510 2023-11-15T04:11:37.068 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.063+0000 7f47e0fc6700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:37.068 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.064+0000 7f47e0fc6700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 0x7f47c8032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:37.068 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.066+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc059100 con 0x7f47dc0f6510 2023-11-15T04:11:37.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.071+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47cc012920 con 0x7f47dc0f6510 2023-11-15T04:11:37.275 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.275+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47dc045ad0 con 0x7f47dc0f6510 2023-11-15T04:11:37.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.275+0000 7f47d1ffb700 1 -- 172.21.15.188:0/3652662883 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47cc00f340 con 0x7f47dc0f6510 2023-11-15T04:11:37.277 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:37.278 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:37.279 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 msgr2=0x7f47c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:37.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 0x7f47c8032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).stop 2023-11-15T04:11:37.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 msgr2=0x7f47dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:37.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f47cc013510 tx=0x7f47cc00b040).stop 2023-11-15T04:11:37.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 shutdown_connections 2023-11-15T04:11:37.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:37.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 --2- 172.21.15.188:0/3652662883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:37.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 >> 172.21.15.188:0/3652662883 conn(0x7f47dc0f1c70 msgr2=0x7f47dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:37.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 shutdown_connections 2023-11-15T04:11:37.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:37.277+0000 7f47e3a2b700 1 -- 172.21.15.188:0/3652662883 wait complete. 2023-11-15T04:11:37.282 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:36.071271+0000 mgr.smithi038.bnbogz (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:38.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:38 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:37.276775+0000 mon.smithi038 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.188:0/3652662883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:38.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:38.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:38.071556+0000 mgr.smithi038.bnbogz (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:39.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.892+0000 7f934fb76700 1 -- 172.21.15.188:0/2192104336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 msgr2=0x7f93480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:39.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.892+0000 7f934fb76700 1 --2- 172.21.15.188:0/2192104336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f93480f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9338009fb0 tx=0x7f9338009b30).stop 2023-11-15T04:11:39.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.892+0000 7f934fb76700 1 -- 172.21.15.188:0/2192104336 shutdown_connections 2023-11-15T04:11:39.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.892+0000 7f934fb76700 1 --2- 172.21.15.188:0/2192104336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f93480f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:39.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.892+0000 7f934fb76700 1 -- 172.21.15.188:0/2192104336 >> 172.21.15.188:0/2192104336 conn(0x7f93480f1c70 msgr2=0x7f93480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:39.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 -- 172.21.15.188:0/2192104336 shutdown_connections 2023-11-15T04:11:39.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 -- 172.21.15.188:0/2192104336 wait complete. 2023-11-15T04:11:39.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 Processor -- start 2023-11-15T04:11:39.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 -- start start 2023-11-15T04:11:39.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:39.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.893+0000 7f934fb76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9348106c20 con 0x7f93480f6510 2023-11-15T04:11:39.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:39.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58194/0 (socket says 172.21.15.188:58194) 2023-11-15T04:11:39.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934d912700 1 -- 172.21.15.188:0/1095076788 learned_addr learned my addr 172.21.15.188:0/1095076788 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:39.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934d912700 1 -- 172.21.15.188:0/1095076788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338005040 con 0x7f93480f6510 2023-11-15T04:11:39.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934d912700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9338013510 tx=0x7f933800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:39.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9338009320 con 0x7f93480f6510 2023-11-15T04:11:39.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9348106e20 con 0x7f93480f6510 2023-11-15T04:11:39.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9338013f00 con 0x7f93480f6510 2023-11-15T04:11:39.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9338014d90 con 0x7f93480f6510 2023-11-15T04:11:39.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.894+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93481072b0 con 0x7f93480f6510 2023-11-15T04:11:39.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.895+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93380095f0 con 0x7f93480f6510 2023-11-15T04:11:39.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.895+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9348045ad0 con 0x7f93480f6510 2023-11-15T04:11:39.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.895+0000 7f933e7fc700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 0x7f9334032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:39.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.895+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9338041000 con 0x7f93480f6510 2023-11-15T04:11:39.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.895+0000 7f934d111700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 0x7f9334032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:39.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.896+0000 7f934d111700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 0x7f9334032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f93440096a0 tx=0x7f9344006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:39.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:39.900+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93380268c0 con 0x7f93480f6510 2023-11-15T04:11:40.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.106+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93480fbf60 con 0x7f93480f6510 2023-11-15T04:11:40.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.107+0000 7f933e7fc700 1 -- 172.21.15.188:0/1095076788 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9338010230 con 0x7f93480f6510 2023-11-15T04:11:40.108 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:40.108 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:40.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 msgr2=0x7f9334032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 0x7f9334032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f93440096a0 tx=0x7f9344006b40).stop 2023-11-15T04:11:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 msgr2=0x7f9348106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9338013510 tx=0x7f933800b040).stop 2023-11-15T04:11:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 shutdown_connections 2023-11-15T04:11:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f933402ff60 0x7f9334032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:40.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 --2- 172.21.15.188:0/1095076788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93480f6510 0x7f9348106730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:40.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 >> 172.21.15.188:0/1095076788 conn(0x7f93480f1c70 msgr2=0x7f93480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:40.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 shutdown_connections 2023-11-15T04:11:40.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:40.109+0000 7f934fb76700 1 -- 172.21.15.188:0/1095076788 wait complete. 2023-11-15T04:11:40.112 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:40.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:40.108514+0000 mon.smithi038 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.188:0/1095076788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:41.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:41.501 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:40.071933+0000 mgr.smithi038.bnbogz (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:42.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 -- 172.21.15.188:0/3970264712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 msgr2=0x7f6f540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:42.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 --2- 172.21.15.188:0/3970264712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f540f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6f44009fb0 tx=0x7f6f44009b30).stop 2023-11-15T04:11:42.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 -- 172.21.15.188:0/3970264712 shutdown_connections 2023-11-15T04:11:42.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 --2- 172.21.15.188:0/3970264712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f540f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:42.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 -- 172.21.15.188:0/3970264712 >> 172.21.15.188:0/3970264712 conn(0x7f6f540f1c70 msgr2=0x7f6f540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:42.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 -- 172.21.15.188:0/3970264712 shutdown_connections 2023-11-15T04:11:42.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.706+0000 7f6f597be700 1 -- 172.21.15.188:0/3970264712 wait complete. 2023-11-15T04:11:42.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f597be700 1 Processor -- start 2023-11-15T04:11:42.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f597be700 1 -- start start 2023-11-15T04:11:42.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f597be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:42.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f597be700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f54106c20 con 0x7f6f540f6510 2023-11-15T04:11:42.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f52ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:42.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f52ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58218/0 (socket says 172.21.15.188:58218) 2023-11-15T04:11:42.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f52ffd700 1 -- 172.21.15.188:0/1906771165 learned_addr learned my addr 172.21.15.188:0/1906771165 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:42.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.707+0000 7f6f52ffd700 1 -- 172.21.15.188:0/1906771165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44005040 con 0x7f6f540f6510 2023-11-15T04:11:42.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f52ffd700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6f44013510 tx=0x7f6f4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:42.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f44009320 con 0x7f6f540f6510 2023-11-15T04:11:42.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f54106e20 con 0x7f6f540f6510 2023-11-15T04:11:42.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f44013f00 con 0x7f6f540f6510 2023-11-15T04:11:42.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f44014d90 con 0x7f6f540f6510 2023-11-15T04:11:42.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.708+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f541072b0 con 0x7f6f540f6510 2023-11-15T04:11:42.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f440095f0 con 0x7f6f540f6510 2023-11-15T04:11:42.714 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f54045ad0 con 0x7f6f540f6510 2023-11-15T04:11:42.714 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f3b7fe700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 0x7f6f3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:42.714 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f44041000 con 0x7f6f540f6510 2023-11-15T04:11:42.714 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f527fc700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 0x7f6f3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:42.715 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.709+0000 7f6f527fc700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 0x7f6f3c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6f480096a0 tx=0x7f6f48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:42.715 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.714+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f4401d7f0 con 0x7f6f540f6510 2023-11-15T04:11:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:42.072303+0000 mgr.smithi038.bnbogz (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:42.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.917+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f540fbf60 con 0x7f6f540f6510 2023-11-15T04:11:42.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.919+0000 7f6f3b7fe700 1 -- 172.21.15.188:0/1906771165 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f44010230 con 0x7f6f540f6510 2023-11-15T04:11:42.919 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:42.919 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:42.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.919+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 msgr2=0x7f6f3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:42.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.919+0000 7f6f597be700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 0x7f6f3c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6f480096a0 tx=0x7f6f48006b40).stop 2023-11-15T04:11:42.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 msgr2=0x7f6f54106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:42.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6f44013510 tx=0x7f6f4400b040).stop 2023-11-15T04:11:42.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 shutdown_connections 2023-11-15T04:11:42.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6f3c02ff60 0x7f6f3c032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:42.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 --2- 172.21.15.188:0/1906771165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f540f6510 0x7f6f54106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:42.922 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 >> 172.21.15.188:0/1906771165 conn(0x7f6f540f1c70 msgr2=0x7f6f540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:42.922 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 shutdown_connections 2023-11-15T04:11:42.922 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:42.920+0000 7f6f597be700 1 -- 172.21.15.188:0/1906771165 wait complete. 2023-11-15T04:11:42.923 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:43.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:43 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:42.919305+0000 mon.smithi038 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.188:0/1906771165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:44.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:44.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:44.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:44 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:44.072690+0000 mgr.smithi038.bnbogz (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:45.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- 172.21.15.188:0/131744662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 msgr2=0x7f5eac0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:45.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/131744662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac0f9190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5ea0009fb0 tx=0x7f5ea0009b30).stop 2023-11-15T04:11:45.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- 172.21.15.188:0/131744662 shutdown_connections 2023-11-15T04:11:45.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/131744662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac0f9190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:45.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- 172.21.15.188:0/131744662 >> 172.21.15.188:0/131744662 conn(0x7f5eac0f0b70 msgr2=0x7f5eac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:45.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- 172.21.15.188:0/131744662 shutdown_connections 2023-11-15T04:11:45.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- 172.21.15.188:0/131744662 wait complete. 2023-11-15T04:11:45.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 Processor -- start 2023-11-15T04:11:45.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.579+0000 7f5eb29c9700 1 -- start start 2023-11-15T04:11:45.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eb29c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:45.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eb29c9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac106bd0 con 0x7f5eac0601e0 2023-11-15T04:11:45.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eabfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:45.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eabfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58240/0 (socket says 172.21.15.188:58240) 2023-11-15T04:11:45.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eabfff700 1 -- 172.21.15.188:0/4246730212 learned_addr learned my addr 172.21.15.188:0/4246730212 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:45.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.580+0000 7f5eabfff700 1 -- 172.21.15.188:0/4246730212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0005040 con 0x7f5eac0601e0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5eabfff700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5ea0004dd0 tx=0x7f5ea0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea0018070 con 0x7f5eac0601e0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac106dd0 con 0x7f5eac0601e0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac107260 con 0x7f5eac0601e0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ea0013410 con 0x7f5eac0601e0 2023-11-15T04:11:45.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.581+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea001c3c0 con 0x7f5eac0601e0 2023-11-15T04:11:45.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.582+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ea001d420 con 0x7f5eac0601e0 2023-11-15T04:11:45.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.582+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c004fa0 con 0x7f5eac0601e0 2023-11-15T04:11:45.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.582+0000 7f5ea8ff9700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 0x7f5e940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:45.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.582+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ea0041660 con 0x7f5eac0601e0 2023-11-15T04:11:45.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.582+0000 7f5eab7fe700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 0x7f5e940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:45.587 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.583+0000 7f5eab7fe700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 0x7f5e940323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5e9c0096a0 tx=0x7f5e9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:45.587 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.587+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ea0027e00 con 0x7f5eac0601e0 2023-11-15T04:11:45.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.792+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e8c005670 con 0x7f5eac0601e0 2023-11-15T04:11:45.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.792+0000 7f5ea8ff9700 1 -- 172.21.15.188:0/4246730212 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ea0027e00 con 0x7f5eac0601e0 2023-11-15T04:11:45.793 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:45.794 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:45.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 msgr2=0x7f5e940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:45.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 0x7f5e940323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5e9c0096a0 tx=0x7f5e9c006b40).stop 2023-11-15T04:11:45.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 msgr2=0x7f5eac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5ea0004dd0 tx=0x7f5ea0009b30).stop 2023-11-15T04:11:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 shutdown_connections 2023-11-15T04:11:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5e9402ff10 0x7f5e940323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 --2- 172.21.15.188:0/4246730212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac0601e0 0x7f5eac1066e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:45.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 >> 172.21.15.188:0/4246730212 conn(0x7f5eac0f0b70 msgr2=0x7f5eac0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:45.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 shutdown_connections 2023-11-15T04:11:45.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:45.794+0000 7f5eb29c9700 1 -- 172.21.15.188:0/4246730212 wait complete. 2023-11-15T04:11:45.797 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:46.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:45.793782+0000 mon.smithi038 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.188:0/4246730212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:47.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:47.184 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:47.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:46 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:46.073033+0000 mgr.smithi038.bnbogz (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:48.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- 172.21.15.188:0/2399105627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 msgr2=0x7eff840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:48.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 --2- 172.21.15.188:0/2399105627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff840f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7eff78009fb0 tx=0x7eff78009b30).stop 2023-11-15T04:11:48.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- 172.21.15.188:0/2399105627 shutdown_connections 2023-11-15T04:11:48.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 --2- 172.21.15.188:0/2399105627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff840f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:48.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- 172.21.15.188:0/2399105627 >> 172.21.15.188:0/2399105627 conn(0x7eff840f1c70 msgr2=0x7eff840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:48.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- 172.21.15.188:0/2399105627 shutdown_connections 2023-11-15T04:11:48.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- 172.21.15.188:0/2399105627 wait complete. 2023-11-15T04:11:48.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 Processor -- start 2023-11-15T04:11:48.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.426+0000 7eff8948e700 1 -- start start 2023-11-15T04:11:48.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff8948e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:48.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff8948e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff84069e20 con 0x7eff840f6510 2023-11-15T04:11:48.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff82ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:48.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff82ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42618/0 (socket says 172.21.15.188:42618) 2023-11-15T04:11:48.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff82ffd700 1 -- 172.21.15.188:0/2008267386 learned_addr learned my addr 172.21.15.188:0/2008267386 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:48.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff82ffd700 1 -- 172.21.15.188:0/2008267386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78005040 con 0x7eff840f6510 2023-11-15T04:11:48.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.427+0000 7eff82ffd700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7eff78013510 tx=0x7eff7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:48.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.428+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff78009320 con 0x7eff840f6510 2023-11-15T04:11:48.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.428+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff84069fc0 con 0x7eff840f6510 2023-11-15T04:11:48.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.428+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8406a450 con 0x7eff840f6510 2023-11-15T04:11:48.433 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.428+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff78013f00 con 0x7eff840f6510 2023-11-15T04:11:48.433 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.428+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff78014d60 con 0x7eff840f6510 2023-11-15T04:11:48.433 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff780095f0 con 0x7eff840f6510 2023-11-15T04:11:48.434 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff64004fa0 con 0x7eff840f6510 2023-11-15T04:11:48.434 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff6b7fe700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 0x7eff6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:48.434 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff78040f30 con 0x7eff840f6510 2023-11-15T04:11:48.435 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff827fc700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 0x7eff6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:48.435 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.429+0000 7eff827fc700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 0x7eff6c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7eff740096a0 tx=0x7eff74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:48.435 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.434+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff7806d2b0 con 0x7eff840f6510 2023-11-15T04:11:48.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.637+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff64005670 con 0x7eff840f6510 2023-11-15T04:11:48.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.638+0000 7eff6b7fe700 1 -- 172.21.15.188:0/2008267386 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff78014420 con 0x7eff840f6510 2023-11-15T04:11:48.639 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:48.639 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:48.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.639+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 msgr2=0x7eff6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:48.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.639+0000 7eff8948e700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 0x7eff6c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7eff740096a0 tx=0x7eff74006b40).stop 2023-11-15T04:11:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.639+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 msgr2=0x7eff8406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.639+0000 7eff8948e700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7eff78013510 tx=0x7eff7800b040).stop 2023-11-15T04:11:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 shutdown_connections 2023-11-15T04:11:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7eff6c02ff60 0x7eff6c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:48.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 --2- 172.21.15.188:0/2008267386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff840f6510 0x7eff8406b9a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:48.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 >> 172.21.15.188:0/2008267386 conn(0x7eff840f1c70 msgr2=0x7eff840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:48.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 shutdown_connections 2023-11-15T04:11:48.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:48.640+0000 7eff8948e700 1 -- 172.21.15.188:0/2008267386 wait complete. 2023-11-15T04:11:48.643 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:48.073275+0000 mgr.smithi038.bnbogz (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:48.639122+0000 mon.smithi038 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.188:0/2008267386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:50.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:50.025 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:51.212 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 -- 172.21.15.188:0/2808801757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 msgr2=0x7fbd6c0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:51.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 --2- 172.21.15.188:0/2808801757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c0f5df0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbd5c009fb0 tx=0x7fbd5c009b30).stop 2023-11-15T04:11:51.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 -- 172.21.15.188:0/2808801757 shutdown_connections 2023-11-15T04:11:51.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 --2- 172.21.15.188:0/2808801757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c0f5df0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:51.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 -- 172.21.15.188:0/2808801757 >> 172.21.15.188:0/2808801757 conn(0x7fbd6c0f02e0 msgr2=0x7fbd6c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:51.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 -- 172.21.15.188:0/2808801757 shutdown_connections 2023-11-15T04:11:51.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.211+0000 7fbd70818700 1 -- 172.21.15.188:0/2808801757 wait complete. 2023-11-15T04:11:51.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd70818700 1 Processor -- start 2023-11-15T04:11:51.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd70818700 1 -- start start 2023-11-15T04:11:51.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd70818700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:51.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd70818700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c106bb0 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd6a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:51.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd6a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42646/0 (socket says 172.21.15.188:42646) 2023-11-15T04:11:51.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.212+0000 7fbd6a59c700 1 -- 172.21.15.188:0/2246973340 learned_addr learned my addr 172.21.15.188:0/2246973340 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:51.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd6a59c700 1 -- 172.21.15.188:0/2246973340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c005040 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd6a59c700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbd5c013510 tx=0x7fbd5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:51.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c009320 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c106db0 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd5c013f00 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c014d90 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.213+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c107240 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.214+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd5c0263a0 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.214+0000 7fbd5affd700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 0x7fbd54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:51.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.214+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd5c040fe0 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.214+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c045ad0 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.214+0000 7fbd69d9b700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 0x7fbd54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:51.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.215+0000 7fbd69d9b700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 0x7fbd54032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:51.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.219+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd5c012920 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.424+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd6c108090 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.425+0000 7fbd5affd700 1 -- 172.21.15.188:0/2246973340 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd5c02b020 con 0x7fbd6c0f59d0 2023-11-15T04:11:51.426 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:51.427 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 msgr2=0x7fbd54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 0x7fbd54032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).stop 2023-11-15T04:11:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 msgr2=0x7fbd6c1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbd5c013510 tx=0x7fbd5c00b040).stop 2023-11-15T04:11:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 shutdown_connections 2023-11-15T04:11:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbd5402ff60 0x7fbd54032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 --2- 172.21.15.188:0/2246973340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd6c0f59d0 0x7fbd6c1066c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 >> 172.21.15.188:0/2246973340 conn(0x7fbd6c0f02e0 msgr2=0x7fbd6c0f4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 shutdown_connections 2023-11-15T04:11:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:51.427+0000 7fbd70818700 1 -- 172.21.15.188:0/2246973340 wait complete. 2023-11-15T04:11:51.430 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:50.073618+0000 mgr.smithi038.bnbogz (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:50.209488+0000 mon.smithi038 (mon.0) 174 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:11:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:50.212098+0000 mon.smithi038 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:11:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:51.426618+0000 mon.smithi038 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.188:0/2246973340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:52.073879+0000 mgr.smithi038.bnbogz (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:52.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:52.853 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:54.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 -- 172.21.15.188:0/3155488120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 msgr2=0x7fe3940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:54.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 --2- 172.21.15.188:0/3155488120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3940f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe384009fb0 tx=0x7fe384009b30).stop 2023-11-15T04:11:54.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 -- 172.21.15.188:0/3155488120 shutdown_connections 2023-11-15T04:11:54.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 --2- 172.21.15.188:0/3155488120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3940f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:54.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 -- 172.21.15.188:0/3155488120 >> 172.21.15.188:0/3155488120 conn(0x7fe3940f0b70 msgr2=0x7fe3940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:54.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 -- 172.21.15.188:0/3155488120 shutdown_connections 2023-11-15T04:11:54.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.007+0000 7fe39ba40700 1 -- 172.21.15.188:0/3155488120 wait complete. 2023-11-15T04:11:54.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe39ba40700 1 Processor -- start 2023-11-15T04:11:54.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe39ba40700 1 -- start start 2023-11-15T04:11:54.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe39ba40700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:54.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe39ba40700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394108db0 con 0x7fe3940f5410 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe3997dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe3997dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42666/0 (socket says 172.21.15.188:42666) 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.008+0000 7fe3997dc700 1 -- 172.21.15.188:0/2183400668 learned_addr learned my addr 172.21.15.188:0/2183400668 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe3997dc700 1 -- 172.21.15.188:0/2183400668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe384005040 con 0x7fe3940f5410 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe3997dc700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe384013510 tx=0x7fe38400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:54.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe384009320 con 0x7fe3940f5410 2023-11-15T04:11:54.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe394108fb0 con 0x7fe3940f5410 2023-11-15T04:11:54.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe384013f00 con 0x7fe3940f5410 2023-11-15T04:11:54.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe384014d90 con 0x7fe3940f5410 2023-11-15T04:11:54.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.009+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe394109440 con 0x7fe3940f5410 2023-11-15T04:11:54.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.010+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3840263a0 con 0x7fe3940f5410 2023-11-15T04:11:54.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.010+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe394045ad0 con 0x7fe3940f5410 2023-11-15T04:11:54.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.010+0000 7fe389ffb700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 0x7fe380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:54.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.010+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3840410b0 con 0x7fe3940f5410 2023-11-15T04:11:54.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.010+0000 7fe398fdb700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 0x7fe380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:54.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.011+0000 7fe398fdb700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 0x7fe380032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe3900096a0 tx=0x7fe390006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:54.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.015+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3840126f0 con 0x7fe3940f5410 2023-11-15T04:11:54.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.225+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe39410a2f0 con 0x7fe3940f5410 2023-11-15T04:11:54.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.226+0000 7fe389ffb700 1 -- 172.21.15.188:0/2183400668 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe38400f340 con 0x7fe3940f5410 2023-11-15T04:11:54.226 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:54.226 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:54.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 msgr2=0x7fe380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:54.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 0x7fe380032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe3900096a0 tx=0x7fe390006b40).stop 2023-11-15T04:11:54.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 msgr2=0x7fe3941088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:54.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe384013510 tx=0x7fe38400b040).stop 2023-11-15T04:11:54.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 shutdown_connections 2023-11-15T04:11:54.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe38002ff60 0x7fe380032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:54.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 --2- 172.21.15.188:0/2183400668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3940f5410 0x7fe3941088c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:54.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 >> 172.21.15.188:0/2183400668 conn(0x7fe3940f0b70 msgr2=0x7fe39405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:54.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 shutdown_connections 2023-11-15T04:11:54.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:54.228+0000 7fe39ba40700 1 -- 172.21.15.188:0/2183400668 wait complete. 2023-11-15T04:11:54.229 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:55.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:54.074176+0000 mgr.smithi038.bnbogz (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:55.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:54.226455+0000 mon.smithi038 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.188:0/2183400668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:55.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:55.614 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:56.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 -- 172.21.15.188:0/1180287900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 msgr2=0x7f7ec40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:56.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1180287900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec40f90f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7eb4009fb0 tx=0x7f7eb4009b30).stop 2023-11-15T04:11:56.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 -- 172.21.15.188:0/1180287900 shutdown_connections 2023-11-15T04:11:56.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1180287900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec40f90f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:56.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 -- 172.21.15.188:0/1180287900 >> 172.21.15.188:0/1180287900 conn(0x7f7ec40f0b70 msgr2=0x7f7ec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:56.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 -- 172.21.15.188:0/1180287900 shutdown_connections 2023-11-15T04:11:56.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.786+0000 7f7ecc514700 1 -- 172.21.15.188:0/1180287900 wait complete. 2023-11-15T04:11:56.789 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7ecc514700 1 Processor -- start 2023-11-15T04:11:56.790 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7ecc514700 1 -- start start 2023-11-15T04:11:56.790 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7ecc514700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:56.790 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7ecc514700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec4106bd0 con 0x7f7ec4060280 2023-11-15T04:11:56.790 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7eca2b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:56.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7eca2b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:42672/0 (socket says 172.21.15.188:42672) 2023-11-15T04:11:56.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.787+0000 7f7eca2b0700 1 -- 172.21.15.188:0/1397454834 learned_addr learned my addr 172.21.15.188:0/1397454834 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:56.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7eca2b0700 1 -- 172.21.15.188:0/1397454834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4005040 con 0x7f7ec4060280 2023-11-15T04:11:56.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7eca2b0700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7eb4013510 tx=0x7f7eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:56.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb4009320 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7eb4013f00 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec4106dd0 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb4014dc0 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.788+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec4107260 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.789+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7eb4014420 con 0x7f7ec4060280 2023-11-15T04:11:56.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.789+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec4045ad0 con 0x7f7ec4060280 2023-11-15T04:11:56.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.789+0000 7f7ebaffd700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 0x7f7eb0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:56.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.789+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7eb4040ff0 con 0x7f7ec4060280 2023-11-15T04:11:56.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.789+0000 7f7ec9aaf700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 0x7f7eb0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:56.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.790+0000 7f7ec9aaf700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 0x7f7eb0032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7ec00096a0 tx=0x7f7ec0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:56.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:56.794+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7eb406d2b0 con 0x7f7ec4060280 2023-11-15T04:11:57.002 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.002+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ec4108110 con 0x7f7ec4060280 2023-11-15T04:11:57.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.003+0000 7f7ebaffd700 1 -- 172.21.15.188:0/1397454834 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7eb40095f0 con 0x7f7ec4060280 2023-11-15T04:11:57.003 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:57.003 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:57.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 msgr2=0x7f7eb0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:57.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 0x7f7eb0032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7ec00096a0 tx=0x7f7ec0006b40).stop 2023-11-15T04:11:57.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 msgr2=0x7f7ec41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:57.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7eb4013510 tx=0x7f7eb400b040).stop 2023-11-15T04:11:57.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 shutdown_connections 2023-11-15T04:11:57.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7eb002ffb0 0x7f7eb0032470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:57.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 --2- 172.21.15.188:0/1397454834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec4060280 0x7f7ec41066e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:57.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 >> 172.21.15.188:0/1397454834 conn(0x7f7ec40f0b70 msgr2=0x7f7ec40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:57.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 shutdown_connections 2023-11-15T04:11:57.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:57.005+0000 7f7ecc514700 1 -- 172.21.15.188:0/1397454834 wait complete. 2023-11-15T04:11:57.007 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:11:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:56.074391+0000 mgr.smithi038.bnbogz (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:57.003511+0000 mon.smithi038 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.188:0/1397454834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:11:58.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:11:58.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:11:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:11:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:11:58.074732+0000 mgr.smithi038.bnbogz (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:11:59.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 -- 172.21.15.188:0/2834882047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 msgr2=0x7f4c600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/2834882047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c600f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f4c50009fb0 tx=0x7f4c50009b30).stop 2023-11-15T04:11:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 -- 172.21.15.188:0/2834882047 shutdown_connections 2023-11-15T04:11:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/2834882047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c600f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 -- 172.21.15.188:0/2834882047 >> 172.21.15.188:0/2834882047 conn(0x7f4c600f0b70 msgr2=0x7f4c600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 -- 172.21.15.188:0/2834882047 shutdown_connections 2023-11-15T04:11:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.701+0000 7f4c65ee1700 1 -- 172.21.15.188:0/2834882047 wait complete. 2023-11-15T04:11:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.702+0000 7f4c65ee1700 1 Processor -- start 2023-11-15T04:11:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.702+0000 7f4c65ee1700 1 -- start start 2023-11-15T04:11:59.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.702+0000 7f4c65ee1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:59.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.702+0000 7f4c65ee1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60060240 con 0x7f4c600f5410 2023-11-15T04:11:59.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:59.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36352/0 (socket says 172.21.15.188:36352) 2023-11-15T04:11:59.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c5f7fe700 1 -- 172.21.15.188:0/3191008324 learned_addr learned my addr 172.21.15.188:0/3191008324 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:11:59.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c5f7fe700 1 -- 172.21.15.188:0/3191008324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50005040 con 0x7f4c600f5410 2023-11-15T04:11:59.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c5f7fe700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4c50014000 tx=0x7f4c5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:59.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c500095f0 con 0x7f4c600f5410 2023-11-15T04:11:59.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60060440 con 0x7f4c600f5410 2023-11-15T04:11:59.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c50007e70 con 0x7f4c600f5410 2023-11-15T04:11:59.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c50014ea0 con 0x7f4c600f5410 2023-11-15T04:11:59.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.703+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c60109ae0 con 0x7f4c600f5410 2023-11-15T04:11:59.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.704+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c500079c0 con 0x7f4c600f5410 2023-11-15T04:11:59.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.704+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c60045ad0 con 0x7f4c600f5410 2023-11-15T04:11:59.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.704+0000 7f4c47fff700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 0x7f4c48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:11:59.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.704+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c50040820 con 0x7f4c600f5410 2023-11-15T04:11:59.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.704+0000 7f4c5effd700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 0x7f4c48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:11:59.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.705+0000 7f4c5effd700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 0x7f4c48032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4c540096a0 tx=0x7f4c54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:11:59.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.709+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c5000f950 con 0x7f4c600f5410 2023-11-15T04:11:59.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.913+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c600fae60 con 0x7f4c600f5410 2023-11-15T04:11:59.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.914+0000 7f4c47fff700 1 -- 172.21.15.188:0/3191008324 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c50010230 con 0x7f4c600f5410 2023-11-15T04:11:59.914 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:11:59.914 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:11:59.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 msgr2=0x7f4c48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:59.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 0x7f4c48032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4c540096a0 tx=0x7f4c54006b40).stop 2023-11-15T04:11:59.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 msgr2=0x7f4c60109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:11:59.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4c50014000 tx=0x7f4c5000b040).stop 2023-11-15T04:11:59.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 shutdown_connections 2023-11-15T04:11:59.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c4802ffb0 0x7f4c48032470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:59.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 --2- 172.21.15.188:0/3191008324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c600f5410 0x7f4c60109430 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:11:59.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 >> 172.21.15.188:0/3191008324 conn(0x7f4c600f0b70 msgr2=0x7f4c600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:11:59.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.915+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 shutdown_connections 2023-11-15T04:11:59.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:11:59.916+0000 7f4c65ee1700 1 -- 172.21.15.188:0/3191008324 wait complete. 2023-11-15T04:11:59.917 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:00.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:00 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:11:59.914006+0000 mon.smithi038 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.188:0/3191008324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:01.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:01.360 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:00.075114+0000 mgr.smithi038.bnbogz (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:02.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 -- 172.21.15.188:0/848027365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 msgr2=0x7f4b1c0f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:02.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 --2- 172.21.15.188:0/848027365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f7f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f4b10009fb0 tx=0x7f4b10009b30).stop 2023-11-15T04:12:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 -- 172.21.15.188:0/848027365 shutdown_connections 2023-11-15T04:12:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 --2- 172.21.15.188:0/848027365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f7f00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 -- 172.21.15.188:0/848027365 >> 172.21.15.188:0/848027365 conn(0x7f4b1c0f0260 msgr2=0x7f4b1c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 -- 172.21.15.188:0/848027365 shutdown_connections 2023-11-15T04:12:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.518+0000 7f4b23522700 1 -- 172.21.15.188:0/848027365 wait complete. 2023-11-15T04:12:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b23522700 1 Processor -- start 2023-11-15T04:12:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b23522700 1 -- start start 2023-11-15T04:12:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b23522700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b23522700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c0f5830 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b212be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b212be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36370/0 (socket says 172.21.15.188:36370) 2023-11-15T04:12:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.519+0000 7f4b212be700 1 -- 172.21.15.188:0/154173840 learned_addr learned my addr 172.21.15.188:0/154173840 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:02.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b212be700 1 -- 172.21.15.188:0/154173840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10005040 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b212be700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4b10014000 tx=0x7f4b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:02.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b100095f0 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c0f59d0 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b10007e70 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b10014ea0 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.520+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c0f5e60 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.521+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b10025070 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.521+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c059100 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.521+0000 7f4b0dffb700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 0x7f4b08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.521+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b10040a30 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.521+0000 7f4b20abd700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 0x7f4b08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.522+0000 7f4b20abd700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 0x7f4b08032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:02.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.526+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b100079c0 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.731+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b1c0fbfc0 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.732+0000 7f4b0dffb700 1 -- 172.21.15.188:0/154173840 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b1000f300 con 0x7f4b1c0f7ae0 2023-11-15T04:12:02.732 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:02.732 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:02.733 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 msgr2=0x7f4b08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:02.733 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 0x7f4b08032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).stop 2023-11-15T04:12:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 msgr2=0x7f4b1c0f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4b10014000 tx=0x7f4b1000b040).stop 2023-11-15T04:12:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 shutdown_connections 2023-11-15T04:12:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b0802ff60 0x7f4b08032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 --2- 172.21.15.188:0/154173840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b1c0f7ae0 0x7f4b1c0f73b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 >> 172.21.15.188:0/154173840 conn(0x7f4b1c0f0260 msgr2=0x7f4b1c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 shutdown_connections 2023-11-15T04:12:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:02.733+0000 7f4b23522700 1 -- 172.21.15.188:0/154173840 wait complete. 2023-11-15T04:12:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:02.075430+0000 mgr.smithi038.bnbogz (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:03.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:02.732227+0000 mon.smithi038 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.188:0/154173840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:04.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:04.142 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:04.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:04.075778+0000 mgr.smithi038.bnbogz (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:05.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 -- 172.21.15.188:0/1187889521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 msgr2=0x7f3cb80f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 --2- 172.21.15.188:0/1187889521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb80f7f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3cac009fb0 tx=0x7f3cac009b30).stop 2023-11-15T04:12:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 -- 172.21.15.188:0/1187889521 shutdown_connections 2023-11-15T04:12:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 --2- 172.21.15.188:0/1187889521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb80f7f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 -- 172.21.15.188:0/1187889521 >> 172.21.15.188:0/1187889521 conn(0x7f3cb80f02e0 msgr2=0x7f3cb80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 -- 172.21.15.188:0/1187889521 shutdown_connections 2023-11-15T04:12:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.384+0000 7f3cbea24700 1 -- 172.21.15.188:0/1187889521 wait complete. 2023-11-15T04:12:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cbea24700 1 Processor -- start 2023-11-15T04:12:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cbea24700 1 -- start start 2023-11-15T04:12:05.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cbea24700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:05.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cbea24700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8106c30 con 0x7f3cb80f7b60 2023-11-15T04:12:05.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36386/0 (socket says 172.21.15.188:36386) 2023-11-15T04:12:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.385+0000 7f3cb7fff700 1 -- 172.21.15.188:0/3155259238 learned_addr learned my addr 172.21.15.188:0/3155259238 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cb7fff700 1 -- 172.21.15.188:0/3155259238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac005040 con 0x7f3cb80f7b60 2023-11-15T04:12:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cb7fff700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f3cac013510 tx=0x7f3cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cac009320 con 0x7f3cb80f7b60 2023-11-15T04:12:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cac013f00 con 0x7f3cb80f7b60 2023-11-15T04:12:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb8106e30 con 0x7f3cb80f7b60 2023-11-15T04:12:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cac014d90 con 0x7f3cb80f7b60 2023-11-15T04:12:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.386+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb81072c0 con 0x7f3cb80f7b60 2023-11-15T04:12:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.387+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cac014420 con 0x7f3cb80f7b60 2023-11-15T04:12:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.387+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb8045ad0 con 0x7f3cb80f7b60 2023-11-15T04:12:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.387+0000 7f3cb4ff9700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.387+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cac0410c0 con 0x7f3cb80f7b60 2023-11-15T04:12:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.388+0000 7f3cb77fe700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:05.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.388+0000 7f3cb77fe700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 0x7f3ca0032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f3ca80096a0 tx=0x7f3ca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:05.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.392+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cac0095f0 con 0x7f3cb80f7b60 2023-11-15T04:12:05.597 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.596+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cb8108170 con 0x7f3cb80f7b60 2023-11-15T04:12:05.598 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.597+0000 7f3cb4ff9700 1 -- 172.21.15.188:0/3155259238 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cac01db70 con 0x7f3cb80f7b60 2023-11-15T04:12:05.598 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:05.599 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:05.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 msgr2=0x7f3ca0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:05.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 0x7f3ca0032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f3ca80096a0 tx=0x7f3ca8006b40).stop 2023-11-15T04:12:05.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 msgr2=0x7f3cb8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f3cac013510 tx=0x7f3cac00b040).stop 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 shutdown_connections 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 --2- 172.21.15.188:0/3155259238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3cb80f7b60 0x7f3cb8106740 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 >> 172.21.15.188:0/3155259238 conn(0x7f3cb80f02e0 msgr2=0x7f3cb80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:05.601 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 shutdown_connections 2023-11-15T04:12:05.602 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:05.599+0000 7f3cbea24700 1 -- 172.21.15.188:0/3155259238 wait complete. 2023-11-15T04:12:05.602 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:06.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:05.598638+0000 mon.smithi038 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.188:0/3155259238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:06.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:06.966 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:07.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:06 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:06.076099+0000 mgr.smithi038.bnbogz (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:08.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/2846650468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 msgr2=0x7f3e680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:08.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/2846650468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e680f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3e5c009fb0 tx=0x7f3e5c009b30).stop 2023-11-15T04:12:08.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/2846650468 shutdown_connections 2023-11-15T04:12:08.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/2846650468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e680f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:08.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/2846650468 >> 172.21.15.188:0/2846650468 conn(0x7f3e680f0b70 msgr2=0x7f3e680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:08.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/2846650468 shutdown_connections 2023-11-15T04:12:08.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/2846650468 wait complete. 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 Processor -- start 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- start start 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.183+0000 7f3e6cf2a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68060280 con 0x7f3e680f5410 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:08.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36392/0 (socket says 172.21.15.188:36392) 2023-11-15T04:12:08.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6659c700 1 -- 172.21.15.188:0/1060274743 learned_addr learned my addr 172.21.15.188:0/1060274743 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:08.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6659c700 1 -- 172.21.15.188:0/1060274743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c005040 con 0x7f3e680f5410 2023-11-15T04:12:08.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6659c700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3e5c013510 tx=0x7f3e5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:08.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e5c009320 con 0x7f3e680f5410 2023-11-15T04:12:08.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e5c013f00 con 0x7f3e680f5410 2023-11-15T04:12:08.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e5c014d90 con 0x7f3e680f5410 2023-11-15T04:12:08.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e68060480 con 0x7f3e680f5410 2023-11-15T04:12:08.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.184+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e68109b20 con 0x7f3e680f5410 2023-11-15T04:12:08.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.185+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e5c0095f0 con 0x7f3e680f5410 2023-11-15T04:12:08.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.185+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e68045ad0 con 0x7f3e680f5410 2023-11-15T04:12:08.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.185+0000 7f3e56ffd700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 0x7f3e50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:08.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.185+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e5c0410e0 con 0x7f3e680f5410 2023-11-15T04:12:08.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.186+0000 7f3e65d9b700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 0x7f3e50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:08.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.186+0000 7f3e65d9b700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 0x7f3e50032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f3e580096a0 tx=0x7f3e58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:08.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.190+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e5c014420 con 0x7f3e680f5410 2023-11-15T04:12:08.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.394+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e6810a340 con 0x7f3e680f5410 2023-11-15T04:12:08.395 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.395+0000 7f3e56ffd700 1 -- 172.21.15.188:0/1060274743 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e5c01d7f0 con 0x7f3e680f5410 2023-11-15T04:12:08.396 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:08.396 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:08.397 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 msgr2=0x7f3e50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:08.397 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 0x7f3e50032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f3e580096a0 tx=0x7f3e58006b40).stop 2023-11-15T04:12:08.398 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 msgr2=0x7f3e681093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:08.398 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3e5c013510 tx=0x7f3e5c00b040).stop 2023-11-15T04:12:08.398 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 shutdown_connections 2023-11-15T04:12:08.398 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3e5002ff60 0x7f3e50032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:08.398 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 --2- 172.21.15.188:0/1060274743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e680f5410 0x7f3e681093f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:08.399 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 >> 172.21.15.188:0/1060274743 conn(0x7f3e680f0b70 msgr2=0x7f3e680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:08.400 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 shutdown_connections 2023-11-15T04:12:08.400 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:08.396+0000 7f3e6cf2a700 1 -- 172.21.15.188:0/1060274743 wait complete. 2023-11-15T04:12:08.401 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:08.076434+0000 mgr.smithi038.bnbogz (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:09.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:08.396085+0000 mon.smithi038 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.188:0/1060274743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:09.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:09.806 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:11.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 -- 172.21.15.188:0/1602122577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 msgr2=0x7f69400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:11.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 --2- 172.21.15.188:0/1602122577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69400f90f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6934009fb0 tx=0x7f6934009b30).stop 2023-11-15T04:12:11.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 -- 172.21.15.188:0/1602122577 shutdown_connections 2023-11-15T04:12:11.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 --2- 172.21.15.188:0/1602122577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69400f90f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:11.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 -- 172.21.15.188:0/1602122577 >> 172.21.15.188:0/1602122577 conn(0x7f69400f0b70 msgr2=0x7f69400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:11.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 -- 172.21.15.188:0/1602122577 shutdown_connections 2023-11-15T04:12:11.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.018+0000 7f694829b700 1 -- 172.21.15.188:0/1602122577 wait complete. 2023-11-15T04:12:11.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.019+0000 7f694829b700 1 Processor -- start 2023-11-15T04:12:11.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.019+0000 7f694829b700 1 -- start start 2023-11-15T04:12:11.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.019+0000 7f694829b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:11.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.019+0000 7f694829b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940106bd0 con 0x7f6940060280 2023-11-15T04:12:11.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6946037700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:11.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6946037700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36526/0 (socket says 172.21.15.188:36526) 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6946037700 1 -- 172.21.15.188:0/1553072984 learned_addr learned my addr 172.21.15.188:0/1553072984 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6946037700 1 -- 172.21.15.188:0/1553072984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934005040 con 0x7f6940060280 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6946037700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6934013510 tx=0x7f693400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6934009320 con 0x7f6940060280 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6940106dd0 con 0x7f6940060280 2023-11-15T04:12:11.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6934013f00 con 0x7f6940060280 2023-11-15T04:12:11.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6934014dc0 con 0x7f6940060280 2023-11-15T04:12:11.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.020+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6940107260 con 0x7f6940060280 2023-11-15T04:12:11.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.021+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69340095f0 con 0x7f6940060280 2023-11-15T04:12:11.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.021+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6940045ad0 con 0x7f6940060280 2023-11-15T04:12:11.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.021+0000 7f6932ffd700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:11.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.021+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6934041040 con 0x7f6940060280 2023-11-15T04:12:11.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.021+0000 7f6945836700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:11.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.022+0000 7f6945836700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 0x7f692c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:11.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.026+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6934014420 con 0x7f6940060280 2023-11-15T04:12:11.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.228+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6940108110 con 0x7f6940060280 2023-11-15T04:12:11.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.228+0000 7f6932ffd700 1 -- 172.21.15.188:0/1553072984 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6934010230 con 0x7f6940060280 2023-11-15T04:12:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:11.230 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:11.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 msgr2=0x7f692c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:11.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 0x7f692c032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).stop 2023-11-15T04:12:11.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 msgr2=0x7f69401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:11.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6934013510 tx=0x7f693400b040).stop 2023-11-15T04:12:11.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 shutdown_connections 2023-11-15T04:12:11.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:11.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 --2- 172.21.15.188:0/1553072984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6940060280 0x7f69401066e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:11.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 >> 172.21.15.188:0/1553072984 conn(0x7f69400f0b70 msgr2=0x7f69400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:11.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 shutdown_connections 2023-11-15T04:12:11.233 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:11.230+0000 7f694829b700 1 -- 172.21.15.188:0/1553072984 wait complete. 2023-11-15T04:12:11.233 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:10.076812+0000 mgr.smithi038.bnbogz (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:12.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:11.229727+0000 mon.smithi038 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.188:0/1553072984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:12.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:12.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:13.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:12.077160+0000 mgr.smithi038.bnbogz (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 -- 172.21.15.188:0/3762767678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 msgr2=0x7f955c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 --2- 172.21.15.188:0/3762767678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c0f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f954c009fb0 tx=0x7f954c009b30).stop 2023-11-15T04:12:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 -- 172.21.15.188:0/3762767678 shutdown_connections 2023-11-15T04:12:13.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 --2- 172.21.15.188:0/3762767678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c0f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:13.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 -- 172.21.15.188:0/3762767678 >> 172.21.15.188:0/3762767678 conn(0x7f955c0f0b70 msgr2=0x7f955c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:13.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 -- 172.21.15.188:0/3762767678 shutdown_connections 2023-11-15T04:12:13.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.890+0000 7f956144c700 1 -- 172.21.15.188:0/3762767678 wait complete. 2023-11-15T04:12:13.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f956144c700 1 Processor -- start 2023-11-15T04:12:13.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f956144c700 1 -- start start 2023-11-15T04:12:13.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f956144c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:13.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f956144c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c060240 con 0x7f955c0f5410 2023-11-15T04:12:13.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f955affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:13.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f955affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36536/0 (socket says 172.21.15.188:36536) 2023-11-15T04:12:13.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.891+0000 7f955affd700 1 -- 172.21.15.188:0/3512004345 learned_addr learned my addr 172.21.15.188:0/3512004345 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:13.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f955affd700 1 -- 172.21.15.188:0/3512004345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c005040 con 0x7f955c0f5410 2023-11-15T04:12:13.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f955affd700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f954c013510 tx=0x7f954c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:13.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954c009320 con 0x7f955c0f5410 2023-11-15T04:12:13.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c060440 con 0x7f955c0f5410 2023-11-15T04:12:13.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f954c013f00 con 0x7f955c0f5410 2023-11-15T04:12:13.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954c014d90 con 0x7f955c0f5410 2023-11-15T04:12:13.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.892+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c109ab0 con 0x7f955c0f5410 2023-11-15T04:12:13.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.893+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f954c0095f0 con 0x7f955c0f5410 2023-11-15T04:12:13.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.893+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c045ad0 con 0x7f955c0f5410 2023-11-15T04:12:13.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.893+0000 7f95537fe700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:13.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.893+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f954c0410c0 con 0x7f955c0f5410 2023-11-15T04:12:13.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.893+0000 7f955a7fc700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:13.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.894+0000 7f955a7fc700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 0x7f9548032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f95440096a0 tx=0x7f9544006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:13.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:13.898+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f954c014420 con 0x7f955c0f5410 2023-11-15T04:12:14.102 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.102+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f955c0fae60 con 0x7f955c0f5410 2023-11-15T04:12:14.103 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.102+0000 7f95537fe700 1 -- 172.21.15.188:0/3512004345 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f954c010230 con 0x7f955c0f5410 2023-11-15T04:12:14.103 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:14.104 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:14.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 msgr2=0x7f9548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:14.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 0x7f9548032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f95440096a0 tx=0x7f9544006b40).stop 2023-11-15T04:12:14.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 msgr2=0x7f955c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:14.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f954c013510 tx=0x7f954c00b040).stop 2023-11-15T04:12:14.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 shutdown_connections 2023-11-15T04:12:14.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:14.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 --2- 172.21.15.188:0/3512004345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f955c0f5410 0x7f955c109400 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:14.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 >> 172.21.15.188:0/3512004345 conn(0x7f955c0f0b70 msgr2=0x7f955c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:14.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 shutdown_connections 2023-11-15T04:12:14.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:14.104+0000 7f956144c700 1 -- 172.21.15.188:0/3512004345 wait complete. 2023-11-15T04:12:14.107 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:14.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:14.103842+0000 mon.smithi038 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.188:0/3512004345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:15.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:15.488 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:15.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:14.077497+0000 mgr.smithi038.bnbogz (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:16.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 -- 172.21.15.188:0/1612931228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 msgr2=0x7f44d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:16.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 --2- 172.21.15.188:0/1612931228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d00f55c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f44c0009fb0 tx=0x7f44c0009b30).stop 2023-11-15T04:12:16.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 -- 172.21.15.188:0/1612931228 shutdown_connections 2023-11-15T04:12:16.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 --2- 172.21.15.188:0/1612931228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d00f55c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:16.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 -- 172.21.15.188:0/1612931228 >> 172.21.15.188:0/1612931228 conn(0x7f44d00f09a0 msgr2=0x7f44d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:16.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 -- 172.21.15.188:0/1612931228 shutdown_connections 2023-11-15T04:12:16.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.697+0000 7f44d708d700 1 -- 172.21.15.188:0/1612931228 wait complete. 2023-11-15T04:12:16.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.698+0000 7f44d708d700 1 Processor -- start 2023-11-15T04:12:16.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.698+0000 7f44d708d700 1 -- start start 2023-11-15T04:12:16.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.698+0000 7f44d708d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:16.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.698+0000 7f44d708d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01058a0 con 0x7f44d00f51a0 2023-11-15T04:12:16.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44d608b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44d608b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36554/0 (socket says 172.21.15.188:36554) 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44d608b700 1 -- 172.21.15.188:0/4175899277 learned_addr learned my addr 172.21.15.188:0/4175899277 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44d608b700 1 -- 172.21.15.188:0/4175899277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0005040 con 0x7f44d00f51a0 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44d608b700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f44c0009830 tx=0x7f44c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.699+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c0009320 con 0x7f44d00f51a0 2023-11-15T04:12:16.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d0105aa0 con 0x7f44d00f51a0 2023-11-15T04:12:16.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44c00136c0 con 0x7f44d00f51a0 2023-11-15T04:12:16.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c001ee90 con 0x7f44d00f51a0 2023-11-15T04:12:16.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d0105f30 con 0x7f44d00f51a0 2023-11-15T04:12:16.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44c001e420 con 0x7f44d00f51a0 2023-11-15T04:12:16.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.700+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0045ad0 con 0x7f44d00f51a0 2023-11-15T04:12:16.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.701+0000 7f44c6ffd700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 0x7f44bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:16.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.701+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44c00410b0 con 0x7f44d00f51a0 2023-11-15T04:12:16.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.701+0000 7f44d588a700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 0x7f44bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:16.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.701+0000 7f44d588a700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 0x7f44bc0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44cc0096a0 tx=0x7f44cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:16.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.706+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44c00095f0 con 0x7f44d00f51a0 2023-11-15T04:12:16.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.908+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44d0106de0 con 0x7f44d00f51a0 2023-11-15T04:12:16.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.909+0000 7f44c6ffd700 1 -- 172.21.15.188:0/4175899277 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44c002c2d0 con 0x7f44d00f51a0 2023-11-15T04:12:16.910 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:16.910 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:16.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 msgr2=0x7f44bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:16.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 0x7f44bc0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44cc0096a0 tx=0x7f44cc006b40).stop 2023-11-15T04:12:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 msgr2=0x7f44d01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f44c0009830 tx=0x7f44c000b040).stop 2023-11-15T04:12:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 shutdown_connections 2023-11-15T04:12:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f44bc02ff10 0x7f44bc0323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:16.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 --2- 172.21.15.188:0/4175899277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d00f51a0 0x7f44d01053b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:16.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.910+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 >> 172.21.15.188:0/4175899277 conn(0x7f44d00f09a0 msgr2=0x7f44d00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:16.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.911+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 shutdown_connections 2023-11-15T04:12:16.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:16.911+0000 7f44d708d700 1 -- 172.21.15.188:0/4175899277 wait complete. 2023-11-15T04:12:16.913 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:16.077797+0000 mgr.smithi038.bnbogz (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:16.910371+0000 mon.smithi038 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.188:0/4175899277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:18.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:18.343 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:19.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/1675867410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 msgr2=0x7f9c980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:19.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/1675867410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c980f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f9c8c009fb0 tx=0x7f9c8c009b30).stop 2023-11-15T04:12:19.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/1675867410 shutdown_connections 2023-11-15T04:12:19.541 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/1675867410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c980f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:19.542 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/1675867410 >> 172.21.15.188:0/1675867410 conn(0x7f9c980f0b70 msgr2=0x7f9c980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:19.542 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/1675867410 shutdown_connections 2023-11-15T04:12:19.542 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.539+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/1675867410 wait complete. 2023-11-15T04:12:19.542 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9eeb9700 1 Processor -- start 2023-11-15T04:12:19.542 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9eeb9700 1 -- start start 2023-11-15T04:12:19.543 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9eeb9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:19.543 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9eeb9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c98106c10 con 0x7f9c980f5410 2023-11-15T04:12:19.544 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9cc55700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:19.545 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9cc55700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45530/0 (socket says 172.21.15.188:45530) 2023-11-15T04:12:19.545 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.540+0000 7f9c9cc55700 1 -- 172.21.15.188:0/932754215 learned_addr learned my addr 172.21.15.188:0/932754215 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:19.545 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c9cc55700 1 -- 172.21.15.188:0/932754215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c005040 con 0x7f9c980f5410 2023-11-15T04:12:19.545 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c9cc55700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f9c8c014000 tx=0x7f9c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:19.546 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c8c0095f0 con 0x7f9c980f5410 2023-11-15T04:12:19.546 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c98106e10 con 0x7f9c980f5410 2023-11-15T04:12:19.546 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c8c007e70 con 0x7f9c980f5410 2023-11-15T04:12:19.546 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c8c014ea0 con 0x7f9c980f5410 2023-11-15T04:12:19.546 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.541+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c981072a0 con 0x7f9c980f5410 2023-11-15T04:12:19.547 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.542+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c8c014690 con 0x7f9c980f5410 2023-11-15T04:12:19.547 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.542+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c98045ad0 con 0x7f9c980f5410 2023-11-15T04:12:19.547 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.542+0000 7f9c957fa700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 0x7f9c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:19.547 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.542+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c8c040930 con 0x7f9c980f5410 2023-11-15T04:12:19.549 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.542+0000 7f9c97fff700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 0x7f9c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:19.549 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.543+0000 7f9c97fff700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 0x7f9c88032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9c840096a0 tx=0x7f9c84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:19.549 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.547+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c8c00f300 con 0x7f9c980f5410 2023-11-15T04:12:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:18.078049+0000 mgr.smithi038.bnbogz (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:19.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.752+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c98108150 con 0x7f9c980f5410 2023-11-15T04:12:19.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.752+0000 7f9c957fa700 1 -- 172.21.15.188:0/932754215 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c8c00f300 con 0x7f9c980f5410 2023-11-15T04:12:19.753 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:19.754 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:19.755 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 msgr2=0x7f9c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:19.755 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 0x7f9c88032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9c840096a0 tx=0x7f9c84006b40).stop 2023-11-15T04:12:19.755 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 msgr2=0x7f9c98106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:19.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f9c8c014000 tx=0x7f9c8c00b040).stop 2023-11-15T04:12:19.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 shutdown_connections 2023-11-15T04:12:19.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9c8802ff60 0x7f9c88032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:19.756 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 --2- 172.21.15.188:0/932754215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c980f5410 0x7f9c98106720 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:19.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 >> 172.21.15.188:0/932754215 conn(0x7f9c980f0b70 msgr2=0x7f9c9805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:19.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 shutdown_connections 2023-11-15T04:12:19.757 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:19.754+0000 7f9c9eeb9700 1 -- 172.21.15.188:0/932754215 wait complete. 2023-11-15T04:12:19.758 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:20.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:20 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:19.753648+0000 mon.smithi038 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.188:0/932754215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:21.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:21.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:20.078303+0000 mgr.smithi038.bnbogz (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:22.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2515236037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 msgr2=0x7fc9ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:22.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2515236037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc994009fb0 tx=0x7fc994009b30).stop 2023-11-15T04:12:22.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2515236037 shutdown_connections 2023-11-15T04:12:22.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2515236037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:22.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2515236037 >> 172.21.15.188:0/2515236037 conn(0x7fc9ac0f0b70 msgr2=0x7fc9ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:22.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2515236037 shutdown_connections 2023-11-15T04:12:22.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.346+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2515236037 wait complete. 2023-11-15T04:12:22.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9b1f11700 1 Processor -- start 2023-11-15T04:12:22.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9b1f11700 1 -- start start 2023-11-15T04:12:22.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9b1f11700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:22.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9b1f11700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac106c00 con 0x7fc9ac0f5410 2023-11-15T04:12:22.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:22.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45562/0 (socket says 172.21.15.188:45562) 2023-11-15T04:12:22.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.347+0000 7fc9ab7fe700 1 -- 172.21.15.188:0/2735294883 learned_addr learned my addr 172.21.15.188:0/2735294883 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:22.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9ab7fe700 1 -- 172.21.15.188:0/2735294883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994005040 con 0x7fc9ac0f5410 2023-11-15T04:12:22.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9ab7fe700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc994013510 tx=0x7fc99400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:22.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc994009320 con 0x7fc9ac0f5410 2023-11-15T04:12:22.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac106e00 con 0x7fc9ac0f5410 2023-11-15T04:12:22.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc994013f00 con 0x7fc9ac0f5410 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc994014d90 con 0x7fc9ac0f5410 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.348+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac107290 con 0x7fc9ac0f5410 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.349+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9940095f0 con 0x7fc9ac0f5410 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.349+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac045ad0 con 0x7fc9ac0f5410 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.349+0000 7fc9a3fff700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 0x7fc998032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:22.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.349+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9940410c0 con 0x7fc9ac0f5410 2023-11-15T04:12:22.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.349+0000 7fc9aaffd700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 0x7fc998032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:22.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.350+0000 7fc9aaffd700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 0x7fc998032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc99c0096a0 tx=0x7fc99c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:22.355 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.354+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc994014420 con 0x7fc9ac0f5410 2023-11-15T04:12:22.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.562+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9ac108140 con 0x7fc9ac0f5410 2023-11-15T04:12:22.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.563+0000 7fc9a3fff700 1 -- 172.21.15.188:0/2735294883 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc994010230 con 0x7fc9ac0f5410 2023-11-15T04:12:22.564 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:22.564 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:22.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 msgr2=0x7fc998032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:22.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 0x7fc998032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc99c0096a0 tx=0x7fc99c006b40).stop 2023-11-15T04:12:22.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 msgr2=0x7fc9ac106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:22.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc994013510 tx=0x7fc99400b040).stop 2023-11-15T04:12:22.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 shutdown_connections 2023-11-15T04:12:22.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc99802ffb0 0x7fc998032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:22.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 --2- 172.21.15.188:0/2735294883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac106710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:22.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.564+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 >> 172.21.15.188:0/2735294883 conn(0x7fc9ac0f0b70 msgr2=0x7fc9ac05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:22.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.565+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 shutdown_connections 2023-11-15T04:12:22.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:22.565+0000 7fc9b1f11700 1 -- 172.21.15.188:0/2735294883 wait complete. 2023-11-15T04:12:22.568 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:22.078583+0000 mgr.smithi038.bnbogz (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:23.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:22.564236+0000 mon.smithi038 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.188:0/2735294883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:23.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:23.964 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:24.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:24 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:24.078901+0000 mgr.smithi038.bnbogz (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:25.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 -- 172.21.15.188:0/3623395265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 msgr2=0x7f2a1c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:25.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 --2- 172.21.15.188:0/3623395265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c0f4be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2a10009fb0 tx=0x7f2a10009b30).stop 2023-11-15T04:12:25.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 -- 172.21.15.188:0/3623395265 shutdown_connections 2023-11-15T04:12:25.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 --2- 172.21.15.188:0/3623395265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c0f4be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:25.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 -- 172.21.15.188:0/3623395265 >> 172.21.15.188:0/3623395265 conn(0x7f2a1c0f02e0 msgr2=0x7f2a1c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:25.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 -- 172.21.15.188:0/3623395265 shutdown_connections 2023-11-15T04:12:25.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.156+0000 7f2a21573700 1 -- 172.21.15.188:0/3623395265 wait complete. 2023-11-15T04:12:25.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a21573700 1 Processor -- start 2023-11-15T04:12:25.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a21573700 1 -- start start 2023-11-15T04:12:25.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a21573700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:25.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a21573700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c106bf0 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a1affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:25.164 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a1affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45572/0 (socket says 172.21.15.188:45572) 2023-11-15T04:12:25.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.157+0000 7f2a1affd700 1 -- 172.21.15.188:0/548286644 learned_addr learned my addr 172.21.15.188:0/548286644 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:25.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a1affd700 1 -- 172.21.15.188:0/548286644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10005040 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a1affd700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2a10013510 tx=0x7f2a1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:25.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a10009320 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c106df0 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a10013f00 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a10014d90 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.158+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c107280 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.159+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c045ad0 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.159+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a100095f0 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.159+0000 7f2a037fe700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 0x7f2a04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:25.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.159+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a10041000 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.160+0000 7f2a1a7fc700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 0x7f2a04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:25.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.160+0000 7f2a1a7fc700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 0x7f2a04032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f2a0c0096a0 tx=0x7f2a0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:25.170 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.164+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a10014420 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.369+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a1c108130 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.369+0000 7f2a037fe700 1 -- 172.21.15.188:0/548286644 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a10010230 con 0x7f2a1c0f47c0 2023-11-15T04:12:25.370 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:25.371 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:25.372 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 msgr2=0x7f2a04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:25.372 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 0x7f2a04032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f2a0c0096a0 tx=0x7f2a0c006b40).stop 2023-11-15T04:12:25.372 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 msgr2=0x7f2a1c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:25.372 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2a10013510 tx=0x7f2a1000b040).stop 2023-11-15T04:12:25.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 shutdown_connections 2023-11-15T04:12:25.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2a0402ff60 0x7f2a04032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:25.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 --2- 172.21.15.188:0/548286644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a1c0f47c0 0x7f2a1c106700 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:25.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 >> 172.21.15.188:0/548286644 conn(0x7f2a1c0f02e0 msgr2=0x7f2a1c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:25.373 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 shutdown_connections 2023-11-15T04:12:25.374 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:25.371+0000 7f2a21573700 1 -- 172.21.15.188:0/548286644 wait complete. 2023-11-15T04:12:25.374 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:25.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:25.370837+0000 mon.smithi038 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.188:0/548286644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:26.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:26.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:26.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:26.079237+0000 mgr.smithi038.bnbogz (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:27.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 -- 172.21.15.188:0/2667756534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 msgr2=0x7fd68c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 --2- 172.21.15.188:0/2667756534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c0f55c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fd67c009fb0 tx=0x7fd67c009b30).stop 2023-11-15T04:12:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 -- 172.21.15.188:0/2667756534 shutdown_connections 2023-11-15T04:12:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 --2- 172.21.15.188:0/2667756534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c0f55c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 -- 172.21.15.188:0/2667756534 >> 172.21.15.188:0/2667756534 conn(0x7fd68c0f09a0 msgr2=0x7fd68c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:27.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 -- 172.21.15.188:0/2667756534 shutdown_connections 2023-11-15T04:12:27.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.962+0000 7fd690b83700 1 -- 172.21.15.188:0/2667756534 wait complete. 2023-11-15T04:12:27.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd690b83700 1 Processor -- start 2023-11-15T04:12:27.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd690b83700 1 -- start start 2023-11-15T04:12:27.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd690b83700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:27.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd690b83700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1069c0 con 0x7fd68c0f51a0 2023-11-15T04:12:27.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd68b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:27.967 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd68b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45586/0 (socket says 172.21.15.188:45586) 2023-11-15T04:12:27.968 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.963+0000 7fd68b7fe700 1 -- 172.21.15.188:0/309579418 learned_addr learned my addr 172.21.15.188:0/309579418 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:27.968 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd68b7fe700 1 -- 172.21.15.188:0/309579418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c005040 con 0x7fd68c0f51a0 2023-11-15T04:12:27.968 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd68b7fe700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd67c013510 tx=0x7fd67c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:27.968 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c009320 con 0x7fd68c0f51a0 2023-11-15T04:12:27.968 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c106bc0 con 0x7fd68c0f51a0 2023-11-15T04:12:27.969 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd67c013f00 con 0x7fd68c0f51a0 2023-11-15T04:12:27.969 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c014dc0 con 0x7fd68c0f51a0 2023-11-15T04:12:27.969 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.964+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c107050 con 0x7fd68c0f51a0 2023-11-15T04:12:27.969 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.965+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd67c0095f0 con 0x7fd68c0f51a0 2023-11-15T04:12:27.970 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.965+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c045ad0 con 0x7fd68c0f51a0 2023-11-15T04:12:27.970 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.965+0000 7fd683fff700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 0x7fd678032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:27.970 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.965+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd67c041100 con 0x7fd68c0f51a0 2023-11-15T04:12:27.970 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.965+0000 7fd68affd700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 0x7fd678032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:27.971 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.966+0000 7fd68affd700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 0x7fd678032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd6740096a0 tx=0x7fd674006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:27.971 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:27.970+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd67c014420 con 0x7fd68c0f51a0 2023-11-15T04:12:28.175 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.174+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd68c107f00 con 0x7fd68c0f51a0 2023-11-15T04:12:28.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.175+0000 7fd683fff700 1 -- 172.21.15.188:0/309579418 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd67c010230 con 0x7fd68c0f51a0 2023-11-15T04:12:28.176 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:28.176 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:28.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 msgr2=0x7fd678032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:28.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 0x7fd678032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd6740096a0 tx=0x7fd674006b40).stop 2023-11-15T04:12:28.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 msgr2=0x7fd68c1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:28.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd67c013510 tx=0x7fd67c00b040).stop 2023-11-15T04:12:28.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 shutdown_connections 2023-11-15T04:12:28.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd67802ff60 0x7fd678032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:28.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 --2- 172.21.15.188:0/309579418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd68c0f51a0 0x7fd68c1064d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:28.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 >> 172.21.15.188:0/309579418 conn(0x7fd68c0f09a0 msgr2=0x7fd68c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:28.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.176+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 shutdown_connections 2023-11-15T04:12:28.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:28.177+0000 7fd690b83700 1 -- 172.21.15.188:0/309579418 wait complete. 2023-11-15T04:12:28.179 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:28.079525+0000 mgr.smithi038.bnbogz (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:29.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:28.176223+0000 mon.smithi038 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.188:0/309579418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:29.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:29.599 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:30.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 -- 172.21.15.188:0/901690150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 msgr2=0x7f63080f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:30.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 --2- 172.21.15.188:0/901690150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63080f90f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f62fc009fb0 tx=0x7f62fc009b30).stop 2023-11-15T04:12:30.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 -- 172.21.15.188:0/901690150 shutdown_connections 2023-11-15T04:12:30.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 --2- 172.21.15.188:0/901690150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63080f90f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:30.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 -- 172.21.15.188:0/901690150 >> 172.21.15.188:0/901690150 conn(0x7f63080f0b70 msgr2=0x7f63080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:30.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 -- 172.21.15.188:0/901690150 shutdown_connections 2023-11-15T04:12:30.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.791+0000 7f630e37f700 1 -- 172.21.15.188:0/901690150 wait complete. 2023-11-15T04:12:30.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f630e37f700 1 Processor -- start 2023-11-15T04:12:30.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f630e37f700 1 -- start start 2023-11-15T04:12:30.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f630e37f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:30.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f630e37f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308106bd0 con 0x7f6308060280 2023-11-15T04:12:30.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f6307fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:30.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f6307fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59068/0 (socket says 172.21.15.188:59068) 2023-11-15T04:12:30.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.792+0000 7f6307fff700 1 -- 172.21.15.188:0/3720221928 learned_addr learned my addr 172.21.15.188:0/3720221928 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:30.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f6307fff700 1 -- 172.21.15.188:0/3720221928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc005040 con 0x7f6308060280 2023-11-15T04:12:30.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f6307fff700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f62fc013510 tx=0x7f62fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:30.797 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc009320 con 0x7f6308060280 2023-11-15T04:12:30.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308106dd0 con 0x7f6308060280 2023-11-15T04:12:30.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62fc013f00 con 0x7f6308060280 2023-11-15T04:12:30.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6308107260 con 0x7f6308060280 2023-11-15T04:12:30.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.793+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc014dc0 con 0x7f6308060280 2023-11-15T04:12:30.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.794+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62fc01d3d0 con 0x7f6308060280 2023-11-15T04:12:30.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.794+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308045ad0 con 0x7f6308060280 2023-11-15T04:12:30.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.794+0000 7f6304ff9700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 0x7f62f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:30.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.794+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62fc041050 con 0x7f6308060280 2023-11-15T04:12:30.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.794+0000 7f63077fe700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 0x7f62f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:30.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.795+0000 7f63077fe700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 0x7f62f0032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:30.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:30.799+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62fc0095f0 con 0x7f6308060280 2023-11-15T04:12:31.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.004+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63081080b0 con 0x7f6308060280 2023-11-15T04:12:31.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.004+0000 7f6304ff9700 1 -- 172.21.15.188:0/3720221928 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62fc01cb10 con 0x7f6308060280 2023-11-15T04:12:31.006 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:31.006 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:31.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 msgr2=0x7f62f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:31.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 0x7f62f0032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).stop 2023-11-15T04:12:31.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 msgr2=0x7f63081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:31.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f62fc013510 tx=0x7f62fc00b040).stop 2023-11-15T04:12:31.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 shutdown_connections 2023-11-15T04:12:31.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f62f002ff60 0x7f62f0032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:31.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 --2- 172.21.15.188:0/3720221928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6308060280 0x7f63081066e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:31.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 >> 172.21.15.188:0/3720221928 conn(0x7f63080f0b70 msgr2=0x7f63080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:31.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 shutdown_connections 2023-11-15T04:12:31.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:31.006+0000 7f630e37f700 1 -- 172.21.15.188:0/3720221928 wait complete. 2023-11-15T04:12:31.009 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:30.079760+0000 mgr.smithi038.bnbogz (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:31.005789+0000 mon.smithi038 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.188:0/3720221928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:32.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:32.424 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:32.080103+0000 mgr.smithi038.bnbogz (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:33.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 -- 172.21.15.188:0/465186647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 msgr2=0x7f67400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:33.641 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 --2- 172.21.15.188:0/465186647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f67400f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f6730009fb0 tx=0x7f6730009b30).stop 2023-11-15T04:12:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 -- 172.21.15.188:0/465186647 shutdown_connections 2023-11-15T04:12:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 --2- 172.21.15.188:0/465186647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f67400f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 -- 172.21.15.188:0/465186647 >> 172.21.15.188:0/465186647 conn(0x7f67400f1c70 msgr2=0x7f67400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 -- 172.21.15.188:0/465186647 shutdown_connections 2023-11-15T04:12:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.639+0000 7f67473c8700 1 -- 172.21.15.188:0/465186647 wait complete. 2023-11-15T04:12:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f67473c8700 1 Processor -- start 2023-11-15T04:12:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f67473c8700 1 -- start start 2023-11-15T04:12:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f67473c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f67473c8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6740069e20 con 0x7f67400f6510 2023-11-15T04:12:33.644 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f6745164700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:33.644 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f6745164700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59088/0 (socket says 172.21.15.188:59088) 2023-11-15T04:12:33.644 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.640+0000 7f6745164700 1 -- 172.21.15.188:0/1141724438 learned_addr learned my addr 172.21.15.188:0/1141724438 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:33.644 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f6745164700 1 -- 172.21.15.188:0/1141724438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730005040 con 0x7f67400f6510 2023-11-15T04:12:33.644 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f6745164700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f6730013510 tx=0x7f673000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:33.646 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6730009320 con 0x7f67400f6510 2023-11-15T04:12:33.646 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740069fc0 con 0x7f67400f6510 2023-11-15T04:12:33.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6730013f00 con 0x7f67400f6510 2023-11-15T04:12:33.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6730014d90 con 0x7f67400f6510 2023-11-15T04:12:33.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.641+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674006a450 con 0x7f67400f6510 2023-11-15T04:12:33.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.642+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6740059100 con 0x7f67400f6510 2023-11-15T04:12:33.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.642+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67300095f0 con 0x7f67400f6510 2023-11-15T04:12:33.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.642+0000 7f6735ffb700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 0x7f672c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:33.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.642+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6730041000 con 0x7f67400f6510 2023-11-15T04:12:33.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.642+0000 7f6744963700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 0x7f672c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:33.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.643+0000 7f6744963700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 0x7f672c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f673c0096a0 tx=0x7f673c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:33.649 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.647+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f673001d7f0 con 0x7f67400f6510 2023-11-15T04:12:33.852 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.852+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67400fbf60 con 0x7f67400f6510 2023-11-15T04:12:33.853 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.852+0000 7f6735ffb700 1 -- 172.21.15.188:0/1141724438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6730010230 con 0x7f67400f6510 2023-11-15T04:12:33.854 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:33.854 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:33.855 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 msgr2=0x7f672c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:33.855 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 0x7f672c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f673c0096a0 tx=0x7f673c006b40).stop 2023-11-15T04:12:33.855 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 msgr2=0x7f674006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:33.856 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f6730013510 tx=0x7f673000b040).stop 2023-11-15T04:12:33.856 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 shutdown_connections 2023-11-15T04:12:33.856 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f672c02ff60 0x7f672c032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:33.856 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 --2- 172.21.15.188:0/1141724438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67400f6510 0x7f674006b9a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:33.856 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 >> 172.21.15.188:0/1141724438 conn(0x7f67400f1c70 msgr2=0x7f67400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:33.857 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 shutdown_connections 2023-11-15T04:12:33.857 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:33.854+0000 7f67473c8700 1 -- 172.21.15.188:0/1141724438 wait complete. 2023-11-15T04:12:33.857 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:34.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:33.853943+0000 mon.smithi038 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.188:0/1141724438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:35.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:35.277 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:35.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:34.080469+0000 mgr.smithi038.bnbogz (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:36.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 -- 172.21.15.188:0/3905226638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 msgr2=0x7f49340f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:36.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 --2- 172.21.15.188:0/3905226638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f49340f8fc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f4924009fb0 tx=0x7f4924009b30).stop 2023-11-15T04:12:36.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 -- 172.21.15.188:0/3905226638 shutdown_connections 2023-11-15T04:12:36.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 --2- 172.21.15.188:0/3905226638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f49340f8fc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:36.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 -- 172.21.15.188:0/3905226638 >> 172.21.15.188:0/3905226638 conn(0x7f49340f09a0 msgr2=0x7f49340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:36.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 -- 172.21.15.188:0/3905226638 shutdown_connections 2023-11-15T04:12:36.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.487+0000 7f493a940700 1 -- 172.21.15.188:0/3905226638 wait complete. 2023-11-15T04:12:36.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493a940700 1 Processor -- start 2023-11-15T04:12:36.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493a940700 1 -- start start 2023-11-15T04:12:36.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493a940700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:36.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493a940700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4934106a10 con 0x7f4934060140 2023-11-15T04:12:36.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493993e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:36.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493993e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59104/0 (socket says 172.21.15.188:59104) 2023-11-15T04:12:36.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.488+0000 7f493993e700 1 -- 172.21.15.188:0/1635049243 learned_addr learned my addr 172.21.15.188:0/1635049243 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:36.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f493993e700 1 -- 172.21.15.188:0/1635049243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924005040 con 0x7f4934060140 2023-11-15T04:12:36.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f493993e700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4924013510 tx=0x7f492400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:36.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4924009320 con 0x7f4934060140 2023-11-15T04:12:36.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4934106c10 con 0x7f4934060140 2023-11-15T04:12:36.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4924013f00 con 0x7f4934060140 2023-11-15T04:12:36.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4924014d90 con 0x7f4934060140 2023-11-15T04:12:36.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.489+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49341070a0 con 0x7f4934060140 2023-11-15T04:12:36.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.490+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49240095f0 con 0x7f4934060140 2023-11-15T04:12:36.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.490+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4934045ad0 con 0x7f4934060140 2023-11-15T04:12:36.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.490+0000 7f492a7fc700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 0x7f4920032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:36.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.490+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49240410c0 con 0x7f4934060140 2023-11-15T04:12:36.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.490+0000 7f493913d700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 0x7f4920032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:36.496 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.491+0000 7f493913d700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 0x7f4920032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f49300096a0 tx=0x7f4930006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:36.496 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.495+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4924014420 con 0x7f4934060140 2023-11-15T04:12:36.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.697+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4934107f50 con 0x7f4934060140 2023-11-15T04:12:36.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.698+0000 7f492a7fc700 1 -- 172.21.15.188:0/1635049243 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4924010230 con 0x7f4934060140 2023-11-15T04:12:36.699 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:36.699 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:36.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 msgr2=0x7f4920032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:36.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 0x7f4920032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f49300096a0 tx=0x7f4930006b40).stop 2023-11-15T04:12:36.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 msgr2=0x7f4934106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:36.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4924013510 tx=0x7f492400b040).stop 2023-11-15T04:12:36.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 shutdown_connections 2023-11-15T04:12:36.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f492002ffb0 0x7f4920032470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:36.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 --2- 172.21.15.188:0/1635049243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934060140 0x7f4934106520 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:36.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 >> 172.21.15.188:0/1635049243 conn(0x7f49340f09a0 msgr2=0x7f49340f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:36.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 shutdown_connections 2023-11-15T04:12:36.702 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:36.699+0000 7f493a940700 1 -- 172.21.15.188:0/1635049243 wait complete. 2023-11-15T04:12:36.702 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:36.080848+0000 mgr.smithi038.bnbogz (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:36.699250+0000 mon.smithi038 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.188:0/1635049243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:38.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:38.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:39.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 -- 172.21.15.188:0/1975729666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 msgr2=0x7fe9d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:39.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/1975729666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d80f6930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe9c0009fb0 tx=0x7fe9c0009b30).stop 2023-11-15T04:12:39.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 -- 172.21.15.188:0/1975729666 shutdown_connections 2023-11-15T04:12:39.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/1975729666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d80f6930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:39.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 -- 172.21.15.188:0/1975729666 >> 172.21.15.188:0/1975729666 conn(0x7fe9d80f1c70 msgr2=0x7fe9d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:39.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 -- 172.21.15.188:0/1975729666 shutdown_connections 2023-11-15T04:12:39.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.259+0000 7fe9dcb11700 1 -- 172.21.15.188:0/1975729666 wait complete. 2023-11-15T04:12:39.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9dcb11700 1 Processor -- start 2023-11-15T04:12:39.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9dcb11700 1 -- start start 2023-11-15T04:12:39.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9dcb11700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:39.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9dcb11700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8106c20 con 0x7fe9d80f6510 2023-11-15T04:12:39.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9d659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:39.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9d659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54508/0 (socket says 172.21.15.188:54508) 2023-11-15T04:12:39.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.260+0000 7fe9d659c700 1 -- 172.21.15.188:0/2774897121 learned_addr learned my addr 172.21.15.188:0/2774897121 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:39.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9d659c700 1 -- 172.21.15.188:0/2774897121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0005040 con 0x7fe9d80f6510 2023-11-15T04:12:39.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9d659c700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe9c0013510 tx=0x7fe9c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:39.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9c0009320 con 0x7fe9d80f6510 2023-11-15T04:12:39.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d8106e20 con 0x7fe9d80f6510 2023-11-15T04:12:39.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c0013f00 con 0x7fe9d80f6510 2023-11-15T04:12:39.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9c0014d90 con 0x7fe9d80f6510 2023-11-15T04:12:39.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.261+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d81072b0 con 0x7fe9d80f6510 2023-11-15T04:12:39.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.262+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8045ad0 con 0x7fe9d80f6510 2023-11-15T04:12:39.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.262+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9c00095f0 con 0x7fe9d80f6510 2023-11-15T04:12:39.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.262+0000 7fe9ceffd700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 0x7fe9c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:39.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.262+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9c0040fe0 con 0x7fe9d80f6510 2023-11-15T04:12:39.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.262+0000 7fe9d5d9b700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 0x7fe9c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:39.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.263+0000 7fe9d5d9b700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 0x7fe9c4032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe9c80096a0 tx=0x7fe9c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:39.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.267+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9c0014420 con 0x7fe9d80f6510 2023-11-15T04:12:39.476 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.475+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9d80fbf60 con 0x7fe9d80f6510 2023-11-15T04:12:39.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.476+0000 7fe9ceffd700 1 -- 172.21.15.188:0/2774897121 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9c0010230 con 0x7fe9d80f6510 2023-11-15T04:12:39.477 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:39.478 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:39.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 msgr2=0x7fe9c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:39.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 0x7fe9c4032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe9c80096a0 tx=0x7fe9c8006b40).stop 2023-11-15T04:12:39.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 msgr2=0x7fe9d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:39.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe9c0013510 tx=0x7fe9c000b040).stop 2023-11-15T04:12:39.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 shutdown_connections 2023-11-15T04:12:39.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe9c402ff60 0x7fe9c4032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:39.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 --2- 172.21.15.188:0/2774897121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9d80f6510 0x7fe9d8106730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:39.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 >> 172.21.15.188:0/2774897121 conn(0x7fe9d80f1c70 msgr2=0x7fe9d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:39.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 shutdown_connections 2023-11-15T04:12:39.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:39.478+0000 7fe9dcb11700 1 -- 172.21.15.188:0/2774897121 wait complete. 2023-11-15T04:12:39.482 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:38.081188+0000 mgr.smithi038.bnbogz (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:40.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:39.477557+0000 mon.smithi038 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.188:0/2774897121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:40.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:40.870 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:40.081507+0000 mgr.smithi038.bnbogz (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:42.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 -- 172.21.15.188:0/1963372751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 msgr2=0x7f55b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:42.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 --2- 172.21.15.188:0/1963372751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b80f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f55a8009fb0 tx=0x7f55a8009b30).stop 2023-11-15T04:12:42.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 -- 172.21.15.188:0/1963372751 shutdown_connections 2023-11-15T04:12:42.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 --2- 172.21.15.188:0/1963372751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b80f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 -- 172.21.15.188:0/1963372751 >> 172.21.15.188:0/1963372751 conn(0x7f55b80f0b70 msgr2=0x7f55b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 -- 172.21.15.188:0/1963372751 shutdown_connections 2023-11-15T04:12:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.101+0000 7f55c06be700 1 -- 172.21.15.188:0/1963372751 wait complete. 2023-11-15T04:12:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55c06be700 1 Processor -- start 2023-11-15T04:12:42.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55c06be700 1 -- start start 2023-11-15T04:12:42.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55c06be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:42.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55c06be700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b8106c00 con 0x7f55b80f5410 2023-11-15T04:12:42.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55be45a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:42.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55be45a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54530/0 (socket says 172.21.15.188:54530) 2023-11-15T04:12:42.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55be45a700 1 -- 172.21.15.188:0/3954564899 learned_addr learned my addr 172.21.15.188:0/3954564899 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:42.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.102+0000 7f55be45a700 1 -- 172.21.15.188:0/3954564899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8005040 con 0x7f55b80f5410 2023-11-15T04:12:42.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55be45a700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f55a8014000 tx=0x7f55a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:42.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a80095f0 con 0x7f55b80f5410 2023-11-15T04:12:42.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b8106e00 con 0x7f55b80f5410 2023-11-15T04:12:42.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55a8007e70 con 0x7f55b80f5410 2023-11-15T04:12:42.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a8014ea0 con 0x7f55b80f5410 2023-11-15T04:12:42.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.103+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b8107290 con 0x7f55b80f5410 2023-11-15T04:12:42.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.104+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b8045ad0 con 0x7f55b80f5410 2023-11-15T04:12:42.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.104+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55a8014690 con 0x7f55b80f5410 2023-11-15T04:12:42.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.104+0000 7f55aeffd700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 0x7f55a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:42.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.104+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55a80408f0 con 0x7f55b80f5410 2023-11-15T04:12:42.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.104+0000 7f55bdc59700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 0x7f55a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:42.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.105+0000 7f55bdc59700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 0x7f55a4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f55b40096a0 tx=0x7f55b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:42.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.110+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55a800f300 con 0x7f55b80f5410 2023-11-15T04:12:42.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.312+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55b80fae60 con 0x7f55b80f5410 2023-11-15T04:12:42.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.313+0000 7f55aeffd700 1 -- 172.21.15.188:0/3954564899 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55a800f300 con 0x7f55b80f5410 2023-11-15T04:12:42.314 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:42.314 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:42.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 msgr2=0x7f55a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:42.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 0x7f55a4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f55b40096a0 tx=0x7f55b4006b40).stop 2023-11-15T04:12:42.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 msgr2=0x7f55b8106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:42.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f55a8014000 tx=0x7f55a800b040).stop 2023-11-15T04:12:42.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 shutdown_connections 2023-11-15T04:12:42.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f55a402ff60 0x7f55a4032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:42.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 --2- 172.21.15.188:0/3954564899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b80f5410 0x7f55b8106710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:42.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 >> 172.21.15.188:0/3954564899 conn(0x7f55b80f0b70 msgr2=0x7f55b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:42.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 shutdown_connections 2023-11-15T04:12:42.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:42.314+0000 7f55c06be700 1 -- 172.21.15.188:0/3954564899 wait complete. 2023-11-15T04:12:42.317 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:42.081858+0000 mgr.smithi038.bnbogz (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:42.314196+0000 mon.smithi038 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.188:0/3954564899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:43.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:43.699 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:44.875 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 -- 172.21.15.188:0/3522201256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 msgr2=0x7f93900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:44.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 --2- 172.21.15.188:0/3522201256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93900f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f9380009fb0 tx=0x7f9380009b30).stop 2023-11-15T04:12:44.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 -- 172.21.15.188:0/3522201256 shutdown_connections 2023-11-15T04:12:44.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 --2- 172.21.15.188:0/3522201256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93900f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:44.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 -- 172.21.15.188:0/3522201256 >> 172.21.15.188:0/3522201256 conn(0x7f93900f0b70 msgr2=0x7f93900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:44.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 -- 172.21.15.188:0/3522201256 shutdown_connections 2023-11-15T04:12:44.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.875+0000 7f9396897700 1 -- 172.21.15.188:0/3522201256 wait complete. 2023-11-15T04:12:44.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.876+0000 7f9396897700 1 Processor -- start 2023-11-15T04:12:44.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.876+0000 7f9396897700 1 -- start start 2023-11-15T04:12:44.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.876+0000 7f9396897700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:44.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.876+0000 7f9396897700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9390060280 con 0x7f93900f5410 2023-11-15T04:12:44.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:44.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54544/0 (socket says 172.21.15.188:54544) 2023-11-15T04:12:44.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938ffff700 1 -- 172.21.15.188:0/1784103931 learned_addr learned my addr 172.21.15.188:0/1784103931 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:44.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938ffff700 1 -- 172.21.15.188:0/1784103931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9380005040 con 0x7f93900f5410 2023-11-15T04:12:44.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938ffff700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9380013510 tx=0x7f938000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:44.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9380009320 con 0x7f93900f5410 2023-11-15T04:12:44.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9390060480 con 0x7f93900f5410 2023-11-15T04:12:44.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9380013f00 con 0x7f93900f5410 2023-11-15T04:12:44.882 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9380014d90 con 0x7f93900f5410 2023-11-15T04:12:44.882 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.877+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9390109af0 con 0x7f93900f5410 2023-11-15T04:12:44.882 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.878+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93800095f0 con 0x7f93900f5410 2023-11-15T04:12:44.883 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.878+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9390045ad0 con 0x7f93900f5410 2023-11-15T04:12:44.883 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.878+0000 7f938cff9700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 0x7f9378032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:44.883 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.878+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9380041080 con 0x7f93900f5410 2023-11-15T04:12:44.884 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.878+0000 7f938f7fe700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 0x7f9378032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:44.884 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.879+0000 7f938f7fe700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 0x7f9378032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f93840096a0 tx=0x7f9384006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:44.885 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:44.883+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9380014420 con 0x7f93900f5410 2023-11-15T04:12:45.088 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.089+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f939010a310 con 0x7f93900f5410 2023-11-15T04:12:45.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.090+0000 7f938cff9700 1 -- 172.21.15.188:0/1784103931 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9380010230 con 0x7f93900f5410 2023-11-15T04:12:45.089 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:45.090 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:45.091 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 msgr2=0x7f9378032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:45.091 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 0x7f9378032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f93840096a0 tx=0x7f9384006b40).stop 2023-11-15T04:12:45.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 msgr2=0x7f93901093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:45.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9380013510 tx=0x7f938000b040).stop 2023-11-15T04:12:45.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 shutdown_connections 2023-11-15T04:12:45.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f937802ff60 0x7f9378032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:45.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 --2- 172.21.15.188:0/1784103931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93900f5410 0x7f93901093c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:45.093 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.091+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 >> 172.21.15.188:0/1784103931 conn(0x7f93900f0b70 msgr2=0x7f93900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:45.093 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.092+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 shutdown_connections 2023-11-15T04:12:45.093 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:45.092+0000 7f9396897700 1 -- 172.21.15.188:0/1784103931 wait complete. 2023-11-15T04:12:45.093 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:44.082122+0000 mgr.smithi038.bnbogz (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:45.089871+0000 mon.smithi038 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.188:0/1784103931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:46.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:46.543 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:46.082406+0000 mgr.smithi038.bnbogz (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 -- 172.21.15.188:0/61291231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 msgr2=0x7fa3a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/61291231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a00f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa394009fb0 tx=0x7fa394009b30).stop 2023-11-15T04:12:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 -- 172.21.15.188:0/61291231 shutdown_connections 2023-11-15T04:12:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/61291231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a00f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 -- 172.21.15.188:0/61291231 >> 172.21.15.188:0/61291231 conn(0x7fa3a00f1c70 msgr2=0x7fa3a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 -- 172.21.15.188:0/61291231 shutdown_connections 2023-11-15T04:12:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 -- 172.21.15.188:0/61291231 wait complete. 2023-11-15T04:12:47.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.684+0000 7fa3a6d72700 1 Processor -- start 2023-11-15T04:12:47.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a6d72700 1 -- start start 2023-11-15T04:12:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a6d72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a6d72700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0106c20 con 0x7fa3a00f6510 2023-11-15T04:12:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a4b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a4b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:54550/0 (socket says 172.21.15.188:54550) 2023-11-15T04:12:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a4b0e700 1 -- 172.21.15.188:0/3418536038 learned_addr learned my addr 172.21.15.188:0/3418536038 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.685+0000 7fa3a4b0e700 1 -- 172.21.15.188:0/3418536038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394005040 con 0x7fa3a00f6510 2023-11-15T04:12:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa3a4b0e700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fa394014000 tx=0x7fa39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3940095f0 con 0x7fa3a00f6510 2023-11-15T04:12:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0106e20 con 0x7fa3a00f6510 2023-11-15T04:12:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa394007e70 con 0x7fa3a00f6510 2023-11-15T04:12:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa394014ea0 con 0x7fa3a00f6510 2023-11-15T04:12:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.686+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a01072b0 con 0x7fa3a00f6510 2023-11-15T04:12:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.687+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa394025070 con 0x7fa3a00f6510 2023-11-15T04:12:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.687+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a0045ad0 con 0x7fa3a00f6510 2023-11-15T04:12:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.687+0000 7fa39d7fa700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:47.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.687+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa394040940 con 0x7fa3a00f6510 2023-11-15T04:12:47.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.687+0000 7fa39ffff700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:47.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.688+0000 7fa39ffff700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 0x7fa390032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:47.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.692+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3940079c0 con 0x7fa3a00f6510 2023-11-15T04:12:47.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.897+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3a00fbf60 con 0x7fa3a00f6510 2023-11-15T04:12:47.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.898+0000 7fa39d7fa700 1 -- 172.21.15.188:0/3418536038 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa39400f300 con 0x7fa3a00f6510 2023-11-15T04:12:47.898 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:47.898 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:47.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 msgr2=0x7fa390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:47.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 0x7fa390032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).stop 2023-11-15T04:12:47.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 msgr2=0x7fa3a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fa394014000 tx=0x7fa39400b040).stop 2023-11-15T04:12:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 shutdown_connections 2023-11-15T04:12:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa39002ff60 0x7fa390032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 --2- 172.21.15.188:0/3418536038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3a00f6510 0x7fa3a0106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 >> 172.21.15.188:0/3418536038 conn(0x7fa3a00f1c70 msgr2=0x7fa3a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 shutdown_connections 2023-11-15T04:12:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:47.899+0000 7fa3a6d72700 1 -- 172.21.15.188:0/3418536038 wait complete. 2023-11-15T04:12:47.902 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:48.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:48 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:47.898170+0000 mon.smithi038 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.188:0/3418536038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:49.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:49.296 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:48.082746+0000 mgr.smithi038.bnbogz (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:50.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 -- 172.21.15.188:0/668343849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 msgr2=0x7fb81c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:50.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 --2- 172.21.15.188:0/668343849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c0fb570 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fb80c009fb0 tx=0x7fb80c009b30).stop 2023-11-15T04:12:50.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 -- 172.21.15.188:0/668343849 shutdown_connections 2023-11-15T04:12:50.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 --2- 172.21.15.188:0/668343849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c0fb570 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 -- 172.21.15.188:0/668343849 >> 172.21.15.188:0/668343849 conn(0x7fb81c0f0b70 msgr2=0x7fb81c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 -- 172.21.15.188:0/668343849 shutdown_connections 2023-11-15T04:12:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.486+0000 7fb822981700 1 -- 172.21.15.188:0/668343849 wait complete. 2023-11-15T04:12:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.487+0000 7fb822981700 1 Processor -- start 2023-11-15T04:12:50.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.487+0000 7fb822981700 1 -- start start 2023-11-15T04:12:50.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.487+0000 7fb822981700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:50.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.487+0000 7fb822981700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c108dc0 con 0x7fb81c060200 2023-11-15T04:12:50.490 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.487+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:50.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55212/0 (socket says 172.21.15.188:55212) 2023-11-15T04:12:50.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb81bfff700 1 -- 172.21.15.188:0/2252828656 learned_addr learned my addr 172.21.15.188:0/2252828656 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:50.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb81bfff700 1 -- 172.21.15.188:0/2252828656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c005040 con 0x7fb81c060200 2023-11-15T04:12:50.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb81bfff700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb80c013510 tx=0x7fb80c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:50.491 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c009320 con 0x7fb81c060200 2023-11-15T04:12:50.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c108fc0 con 0x7fb81c060200 2023-11-15T04:12:50.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb80c013f00 con 0x7fb81c060200 2023-11-15T04:12:50.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c014dc0 con 0x7fb81c060200 2023-11-15T04:12:50.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.488+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c109450 con 0x7fb81c060200 2023-11-15T04:12:50.492 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.489+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c045ad0 con 0x7fb81c060200 2023-11-15T04:12:50.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.489+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb80c014420 con 0x7fb81c060200 2023-11-15T04:12:50.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.489+0000 7fb818ff9700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 0x7fb804032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:50.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.489+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb80c0410e0 con 0x7fb81c060200 2023-11-15T04:12:50.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.489+0000 7fb81b7fe700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 0x7fb804032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:50.493 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.490+0000 7fb81b7fe700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 0x7fb804032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:50.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.494+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb80c0095f0 con 0x7fb81c060200 2023-11-15T04:12:50.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.701+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb81c10a2a0 con 0x7fb81c060200 2023-11-15T04:12:50.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.702+0000 7fb818ff9700 1 -- 172.21.15.188:0/2252828656 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb80c01cb10 con 0x7fb81c060200 2023-11-15T04:12:50.702 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:50.702 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:50.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 msgr2=0x7fb804032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:50.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 0x7fb804032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).stop 2023-11-15T04:12:50.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 msgr2=0x7fb81c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:50.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb80c013510 tx=0x7fb80c00b040).stop 2023-11-15T04:12:50.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 shutdown_connections 2023-11-15T04:12:50.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb80402ff60 0x7fb804032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:50.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 --2- 172.21.15.188:0/2252828656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb81c060200 0x7fb81c1088d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 >> 172.21.15.188:0/2252828656 conn(0x7fb81c0f0b70 msgr2=0x7fb81c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 shutdown_connections 2023-11-15T04:12:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:50.703+0000 7fb822981700 1 -- 172.21.15.188:0/2252828656 wait complete. 2023-11-15T04:12:50.705 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:50.083034+0000 mgr.smithi038.bnbogz (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:50.210629+0000 mon.smithi038 (mon.0) 197 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:12:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:50.212974+0000 mon.smithi038 (mon.0) 198 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:12:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:50.702143+0000 mon.smithi038 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.188:0/2252828656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:52.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:52.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:52.083312+0000 mgr.smithi038.bnbogz (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:53.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 -- 172.21.15.188:0/391862944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 msgr2=0x7fbd000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:53.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 --2- 172.21.15.188:0/391862944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd000f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fbcf0009fb0 tx=0x7fbcf0009b30).stop 2023-11-15T04:12:53.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 -- 172.21.15.188:0/391862944 shutdown_connections 2023-11-15T04:12:53.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 --2- 172.21.15.188:0/391862944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd000f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:53.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 -- 172.21.15.188:0/391862944 >> 172.21.15.188:0/391862944 conn(0x7fbd000f1c70 msgr2=0x7fbd000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:53.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 -- 172.21.15.188:0/391862944 shutdown_connections 2023-11-15T04:12:53.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.253+0000 7fbd0742c700 1 -- 172.21.15.188:0/391862944 wait complete. 2023-11-15T04:12:53.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.254+0000 7fbd0742c700 1 Processor -- start 2023-11-15T04:12:53.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.254+0000 7fbd0742c700 1 -- start start 2023-11-15T04:12:53.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.254+0000 7fbd0742c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:53.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.254+0000 7fbd0742c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd00106c20 con 0x7fbd000f6510 2023-11-15T04:12:53.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.254+0000 7fbd051c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:53.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd051c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55226/0 (socket says 172.21.15.188:55226) 2023-11-15T04:12:53.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd051c8700 1 -- 172.21.15.188:0/2747464067 learned_addr learned my addr 172.21.15.188:0/2747464067 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:53.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd051c8700 1 -- 172.21.15.188:0/2747464067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf0005040 con 0x7fbd000f6510 2023-11-15T04:12:53.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd051c8700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbcf0013510 tx=0x7fbcf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:53.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0009320 con 0x7fbd000f6510 2023-11-15T04:12:53.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcf0013f00 con 0x7fbd000f6510 2023-11-15T04:12:53.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd00106e20 con 0x7fbd000f6510 2023-11-15T04:12:53.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0014d90 con 0x7fbd000f6510 2023-11-15T04:12:53.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.255+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd001072b0 con 0x7fbd000f6510 2023-11-15T04:12:53.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.256+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcf00095f0 con 0x7fbd000f6510 2023-11-15T04:12:53.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.256+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd00045ad0 con 0x7fbd000f6510 2023-11-15T04:12:53.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.256+0000 7fbcf5ffb700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:53.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.256+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcf00410c0 con 0x7fbd000f6510 2023-11-15T04:12:53.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.257+0000 7fbd049c7700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:53.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.257+0000 7fbd049c7700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 0x7fbcec032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbcfc0096a0 tx=0x7fbcfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:53.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.261+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcf0014420 con 0x7fbd000f6510 2023-11-15T04:12:53.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.468+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd000fbf60 con 0x7fbd000f6510 2023-11-15T04:12:53.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.468+0000 7fbcf5ffb700 1 -- 172.21.15.188:0/2747464067 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcf001d7f0 con 0x7fbd000f6510 2023-11-15T04:12:53.468 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:53.469 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:53.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 msgr2=0x7fbcec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:53.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 0x7fbcec032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbcfc0096a0 tx=0x7fbcfc006b40).stop 2023-11-15T04:12:53.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 msgr2=0x7fbd00106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:53.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbcf0013510 tx=0x7fbcf000b040).stop 2023-11-15T04:12:53.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 shutdown_connections 2023-11-15T04:12:53.471 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:53.471 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 --2- 172.21.15.188:0/2747464067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd000f6510 0x7fbd00106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:53.471 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 >> 172.21.15.188:0/2747464067 conn(0x7fbd000f1c70 msgr2=0x7fbd000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:53.471 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 shutdown_connections 2023-11-15T04:12:53.471 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:53.470+0000 7fbd0742c700 1 -- 172.21.15.188:0/2747464067 wait complete. 2023-11-15T04:12:53.472 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:53.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:53.468662+0000 mon.smithi038 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.188:0/2747464067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:54.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:54.083684+0000 mgr.smithi038.bnbogz (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:54.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:54.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:56.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 -- 172.21.15.188:0/2935429317 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 msgr2=0x7f381c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:56.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 --2- 172.21.15.188:0/2935429317 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c0f6930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3810009fb0 tx=0x7f3810009b30).stop 2023-11-15T04:12:56.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 -- 172.21.15.188:0/2935429317 shutdown_connections 2023-11-15T04:12:56.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 --2- 172.21.15.188:0/2935429317 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c0f6930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:56.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 -- 172.21.15.188:0/2935429317 >> 172.21.15.188:0/2935429317 conn(0x7f381c0f1c70 msgr2=0x7f381c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:56.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 -- 172.21.15.188:0/2935429317 shutdown_connections 2023-11-15T04:12:56.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.046+0000 7f3823d77700 1 -- 172.21.15.188:0/2935429317 wait complete. 2023-11-15T04:12:56.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3823d77700 1 Processor -- start 2023-11-15T04:12:56.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3823d77700 1 -- start start 2023-11-15T04:12:56.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3823d77700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:56.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3823d77700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c069e20 con 0x7f381c0f6510 2023-11-15T04:12:56.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3821b13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:56.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3821b13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55252/0 (socket says 172.21.15.188:55252) 2023-11-15T04:12:56.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.047+0000 7f3821b13700 1 -- 172.21.15.188:0/2736016192 learned_addr learned my addr 172.21.15.188:0/2736016192 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:56.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f3821b13700 1 -- 172.21.15.188:0/2736016192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810005040 con 0x7f381c0f6510 2023-11-15T04:12:56.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f3821b13700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3810013510 tx=0x7f381000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:56.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3810009320 con 0x7f381c0f6510 2023-11-15T04:12:56.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3810013f00 con 0x7f381c0f6510 2023-11-15T04:12:56.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c069fc0 con 0x7f381c0f6510 2023-11-15T04:12:56.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3810014d90 con 0x7f381c0f6510 2023-11-15T04:12:56.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.048+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c06a450 con 0x7f381c0f6510 2023-11-15T04:12:56.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38100095f0 con 0x7f381c0f6510 2023-11-15T04:12:56.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c059100 con 0x7f381c0f6510 2023-11-15T04:12:56.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f380e7fc700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 0x7f3808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:56.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3810041000 con 0x7f381c0f6510 2023-11-15T04:12:56.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f3821312700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 0x7f3808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:56.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.049+0000 7f3821312700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 0x7f3808032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f38180096a0 tx=0x7f3818006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:56.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.054+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3810014420 con 0x7f381c0f6510 2023-11-15T04:12:56.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.258+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f381c0fbf60 con 0x7f381c0f6510 2023-11-15T04:12:56.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.259+0000 7f380e7fc700 1 -- 172.21.15.188:0/2736016192 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f381001d7f0 con 0x7f381c0f6510 2023-11-15T04:12:56.260 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:56.260 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:56.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 msgr2=0x7f3808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:56.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 0x7f3808032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f38180096a0 tx=0x7f3818006b40).stop 2023-11-15T04:12:56.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 msgr2=0x7f381c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:56.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3810013510 tx=0x7f381000b040).stop 2023-11-15T04:12:56.262 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 shutdown_connections 2023-11-15T04:12:56.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f380802ff60 0x7f3808032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:56.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 --2- 172.21.15.188:0/2736016192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c0f6510 0x7f381c06b9a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:56.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 >> 172.21.15.188:0/2736016192 conn(0x7f381c0f1c70 msgr2=0x7f381c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:56.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 shutdown_connections 2023-11-15T04:12:56.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:56.261+0000 7f3823d77700 1 -- 172.21.15.188:0/2736016192 wait complete. 2023-11-15T04:12:56.264 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:56.083968+0000 mgr.smithi038.bnbogz (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:56.260510+0000 mon.smithi038 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.188:0/2736016192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:12:57.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:12:57.650 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:12:58.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 -- 172.21.15.188:0/4202006790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 msgr2=0x7fafe80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:58.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 --2- 172.21.15.188:0/4202006790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fafdc009fb0 tx=0x7fafdc009b30).stop 2023-11-15T04:12:58.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 -- 172.21.15.188:0/4202006790 shutdown_connections 2023-11-15T04:12:58.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 --2- 172.21.15.188:0/4202006790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:58.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 -- 172.21.15.188:0/4202006790 >> 172.21.15.188:0/4202006790 conn(0x7fafe80f0b70 msgr2=0x7fafe80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:58.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 -- 172.21.15.188:0/4202006790 shutdown_connections 2023-11-15T04:12:58.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.867+0000 7fafef141700 1 -- 172.21.15.188:0/4202006790 wait complete. 2023-11-15T04:12:58.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafef141700 1 Processor -- start 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafef141700 1 -- start start 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafef141700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafef141700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80fb820 con 0x7fafe80f5410 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafecedd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafecedd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39156/0 (socket says 172.21.15.188:39156) 2023-11-15T04:12:58.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.868+0000 7fafecedd700 1 -- 172.21.15.188:0/717221253 learned_addr learned my addr 172.21.15.188:0/717221253 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:12:58.873 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafecedd700 1 -- 172.21.15.188:0/717221253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafe80f5410 2023-11-15T04:12:58.873 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafecedd700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fafdc009f80 tx=0x7fafdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:58.874 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc009320 con 0x7fafe80f5410 2023-11-15T04:12:58.874 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe80fb9c0 con 0x7fafe80f5410 2023-11-15T04:12:58.874 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafdc013ec0 con 0x7fafe80f5410 2023-11-15T04:12:58.874 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc01ece0 con 0x7fafe80f5410 2023-11-15T04:12:58.875 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.869+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe80fbe50 con 0x7fafe80f5410 2023-11-15T04:12:58.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.870+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafdc01e420 con 0x7fafe80f5410 2023-11-15T04:12:58.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.870+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8059100 con 0x7fafe80f5410 2023-11-15T04:12:58.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.870+0000 7fafe57fa700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 0x7fafd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:12:58.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.870+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafdc040ee0 con 0x7fafe80f5410 2023-11-15T04:12:58.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.870+0000 7fafe7fff700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 0x7fafd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:12:58.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.871+0000 7fafe7fff700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 0x7fafd8032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fafd40096a0 tx=0x7fafd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:12:58.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:58.875+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafdc01e730 con 0x7fafe80f5410 2023-11-15T04:12:59.079 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.078+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafe8045ad0 con 0x7fafe80f5410 2023-11-15T04:12:59.079 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.079+0000 7fafe57fa700 1 -- 172.21.15.188:0/717221253 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafdc01cb10 con 0x7fafe80f5410 2023-11-15T04:12:59.080 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:12:59.080 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:12:59.081 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 msgr2=0x7fafd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:59.081 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 0x7fafd8032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fafd40096a0 tx=0x7fafd4006b40).stop 2023-11-15T04:12:59.081 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 msgr2=0x7fafe80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:12:59.081 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fafdc009f80 tx=0x7fafdc00b040).stop 2023-11-15T04:12:59.082 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 shutdown_connections 2023-11-15T04:12:59.082 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fafd802ff60 0x7fafd8032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:59.082 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 --2- 172.21.15.188:0/717221253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafe80f5410 0x7fafe80fd3a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:12:59.082 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 >> 172.21.15.188:0/717221253 conn(0x7fafe80f0b70 msgr2=0x7fafe805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:12:59.083 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 shutdown_connections 2023-11-15T04:12:59.083 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:12:59.080+0000 7fafef141700 1 -- 172.21.15.188:0/717221253 wait complete. 2023-11-15T04:12:59.083 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:12:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:12:58.084179+0000 mgr.smithi038.bnbogz (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:12:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:12:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:12:59.080095+0000 mon.smithi038 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.188:0/717221253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:00.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:00.486 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:00.084438+0000 mgr.smithi038.bnbogz (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:01.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- 172.21.15.188:0/1640606348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 msgr2=0x7fd7b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:01.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1640606348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b40f55c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd7b0009fb0 tx=0x7fd7b0009b30).stop 2023-11-15T04:13:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- 172.21.15.188:0/1640606348 shutdown_connections 2023-11-15T04:13:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1640606348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b40f55c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- 172.21.15.188:0/1640606348 >> 172.21.15.188:0/1640606348 conn(0x7fd7b40f09a0 msgr2=0x7fd7b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- 172.21.15.188:0/1640606348 shutdown_connections 2023-11-15T04:13:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- 172.21.15.188:0/1640606348 wait complete. 2023-11-15T04:13:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 Processor -- start 2023-11-15T04:13:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.646+0000 7fd7ba120700 1 -- start start 2023-11-15T04:13:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7ba120700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7ba120700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b41058a0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7b911e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7b911e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39172/0 (socket says 172.21.15.188:39172) 2023-11-15T04:13:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7b911e700 1 -- 172.21.15.188:0/1951935601 learned_addr learned my addr 172.21.15.188:0/1951935601 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7b911e700 1 -- 172.21.15.188:0/1951935601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0005040 con 0x7fd7b40f51a0 2023-11-15T04:13:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7b911e700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0009320 con 0x7fd7b40f51a0 2023-11-15T04:13:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b4105aa0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7b0013f00 con 0x7fd7b40f51a0 2023-11-15T04:13:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.647+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0014dc0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.648+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b4105f30 con 0x7fd7b40f51a0 2023-11-15T04:13:01.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.648+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7b00095f0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.648+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b4045ad0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.649+0000 7fd7a9ffb700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.649+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7b0041040 con 0x7fd7b40f51a0 2023-11-15T04:13:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.649+0000 7fd7b891d700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:01.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.649+0000 7fd7b891d700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:01.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.654+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7b0014420 con 0x7fd7b40f51a0 2023-11-15T04:13:01.859 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.858+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7b4106de0 con 0x7fd7b40f51a0 2023-11-15T04:13:01.859 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.859+0000 7fd7a9ffb700 1 -- 172.21.15.188:0/1951935601 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd7b0010230 con 0x7fd7b40f51a0 2023-11-15T04:13:01.861 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:01.861 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:01.862 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 msgr2=0x7fd7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:01.862 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).stop 2023-11-15T04:13:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 msgr2=0x7fd7b41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).stop 2023-11-15T04:13:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 shutdown_connections 2023-11-15T04:13:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 --2- 172.21.15.188:0/1951935601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7b40f51a0 0x7fd7b41053b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:01.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 >> 172.21.15.188:0/1951935601 conn(0x7fd7b40f09a0 msgr2=0x7fd7b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:01.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 shutdown_connections 2023-11-15T04:13:01.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:01.860+0000 7fd7ba120700 1 -- 172.21.15.188:0/1951935601 wait complete. 2023-11-15T04:13:01.864 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:01.859981+0000 mon.smithi038 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.188:0/1951935601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:02.084719+0000 mgr.smithi038.bnbogz (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:03.285 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:03.286 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:04.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.435+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2727003332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 msgr2=0x7f59d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:04.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.435+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2727003332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d00f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f59c0009fb0 tx=0x7f59c0009b30).stop 2023-11-15T04:13:04.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.435+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2727003332 shutdown_connections 2023-11-15T04:13:04.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.435+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2727003332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d00f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:04.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.435+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2727003332 >> 172.21.15.188:0/2727003332 conn(0x7f59d00f0b70 msgr2=0x7f59d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:04.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2727003332 shutdown_connections 2023-11-15T04:13:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2727003332 wait complete. 2023-11-15T04:13:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 Processor -- start 2023-11-15T04:13:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 -- start start 2023-11-15T04:13:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.436+0000 7f59d5ee0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0060220 con 0x7f59d00f5410 2023-11-15T04:13:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59cf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59cf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39180/0 (socket says 172.21.15.188:39180) 2023-11-15T04:13:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59cf7fe700 1 -- 172.21.15.188:0/2605121037 learned_addr learned my addr 172.21.15.188:0/2605121037 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59cf7fe700 1 -- 172.21.15.188:0/2605121037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0005040 con 0x7f59d00f5410 2023-11-15T04:13:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59cf7fe700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f59c0013510 tx=0x7f59c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c0009320 con 0x7f59d00f5410 2023-11-15T04:13:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d0060420 con 0x7f59d00f5410 2023-11-15T04:13:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59c0013f00 con 0x7f59d00f5410 2023-11-15T04:13:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c0014d90 con 0x7f59d00f5410 2023-11-15T04:13:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.437+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d0109b00 con 0x7f59d00f5410 2023-11-15T04:13:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.438+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59c00263a0 con 0x7f59d00f5410 2023-11-15T04:13:04.444 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.438+0000 7f59b7fff700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.438+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59c00410e0 con 0x7f59d00f5410 2023-11-15T04:13:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.438+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0059100 con 0x7f59d00f5410 2023-11-15T04:13:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.439+0000 7f59ceffd700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.439+0000 7f59ceffd700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 0x7f59b8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:04.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.443+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59c0012920 con 0x7f59d00f5410 2023-11-15T04:13:04.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.650+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59d0045ad0 con 0x7f59d00f5410 2023-11-15T04:13:04.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.650+0000 7f59b7fff700 1 -- 172.21.15.188:0/2605121037 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59c002b020 con 0x7f59d00f5410 2023-11-15T04:13:04.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:04.652 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 msgr2=0x7f59b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 0x7f59b8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).stop 2023-11-15T04:13:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 msgr2=0x7f59d0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f59c0013510 tx=0x7f59c000b040).stop 2023-11-15T04:13:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 shutdown_connections 2023-11-15T04:13:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 --2- 172.21.15.188:0/2605121037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59d00f5410 0x7f59d0109450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 >> 172.21.15.188:0/2605121037 conn(0x7f59d00f0b70 msgr2=0x7f59d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 shutdown_connections 2023-11-15T04:13:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:04.652+0000 7f59d5ee0700 1 -- 172.21.15.188:0/2605121037 wait complete. 2023-11-15T04:13:04.655 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:04.085043+0000 mgr.smithi038.bnbogz (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:04.651720+0000 mon.smithi038 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.188:0/2605121037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:06.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:06.045 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:07.263 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 -- 172.21.15.188:0/4238568426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 msgr2=0x7f5d200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:07.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 --2- 172.21.15.188:0/4238568426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d200f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f5d14009fb0 tx=0x7f5d14009b30).stop 2023-11-15T04:13:07.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 -- 172.21.15.188:0/4238568426 shutdown_connections 2023-11-15T04:13:07.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 --2- 172.21.15.188:0/4238568426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d200f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:07.264 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 -- 172.21.15.188:0/4238568426 >> 172.21.15.188:0/4238568426 conn(0x7f5d200f0b70 msgr2=0x7f5d200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:07.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 -- 172.21.15.188:0/4238568426 shutdown_connections 2023-11-15T04:13:07.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.261+0000 7f5d279c6700 1 -- 172.21.15.188:0/4238568426 wait complete. 2023-11-15T04:13:07.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d279c6700 1 Processor -- start 2023-11-15T04:13:07.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d279c6700 1 -- start start 2023-11-15T04:13:07.265 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d279c6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:07.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d279c6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20107210 con 0x7f5d200f5410 2023-11-15T04:13:07.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d25762700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:07.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d25762700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39202/0 (socket says 172.21.15.188:39202) 2023-11-15T04:13:07.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.262+0000 7f5d25762700 1 -- 172.21.15.188:0/3998112714 learned_addr learned my addr 172.21.15.188:0/3998112714 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:07.266 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d25762700 1 -- 172.21.15.188:0/3998112714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d14005040 con 0x7f5d200f5410 2023-11-15T04:13:07.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d25762700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5d14013510 tx=0x7f5d1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:07.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d14009320 con 0x7f5d200f5410 2023-11-15T04:13:07.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d20060220 con 0x7f5d200f5410 2023-11-15T04:13:07.267 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d14013f00 con 0x7f5d200f5410 2023-11-15T04:13:07.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d14014d90 con 0x7f5d200f5410 2023-11-15T04:13:07.268 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.263+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2010bd10 con 0x7f5d200f5410 2023-11-15T04:13:07.269 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.264+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d20045ad0 con 0x7f5d200f5410 2023-11-15T04:13:07.269 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.264+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d140095f0 con 0x7f5d200f5410 2023-11-15T04:13:07.269 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.264+0000 7f5d11ffb700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 0x7f5d0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:07.270 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.264+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d14041080 con 0x7f5d200f5410 2023-11-15T04:13:07.270 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.264+0000 7f5d24f61700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 0x7f5d0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:07.270 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.265+0000 7f5d24f61700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 0x7f5d0c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:07.270 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.269+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d14014420 con 0x7f5d200f5410 2023-11-15T04:13:07.474 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.473+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d200fae60 con 0x7f5d200f5410 2023-11-15T04:13:07.475 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.474+0000 7f5d11ffb700 1 -- 172.21.15.188:0/3998112714 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d14010230 con 0x7f5d200f5410 2023-11-15T04:13:07.475 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:07.475 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:07.476 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 msgr2=0x7f5d0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:07.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 0x7f5d0c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).stop 2023-11-15T04:13:07.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 msgr2=0x7f5d2010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:07.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5d14013510 tx=0x7f5d1400b040).stop 2023-11-15T04:13:07.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 shutdown_connections 2023-11-15T04:13:07.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d0c02ff60 0x7f5d0c032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:07.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 --2- 172.21.15.188:0/3998112714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d200f5410 0x7f5d2010b820 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:07.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 >> 172.21.15.188:0/3998112714 conn(0x7f5d200f0b70 msgr2=0x7f5d200f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:07.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 shutdown_connections 2023-11-15T04:13:07.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:07.476+0000 7f5d279c6700 1 -- 172.21.15.188:0/3998112714 wait complete. 2023-11-15T04:13:07.479 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:06.085350+0000 mgr.smithi038.bnbogz (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:08.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:08 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:07.475582+0000 mon.smithi038 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.188:0/3998112714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:08.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:08.877 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:08.085689+0000 mgr.smithi038.bnbogz (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:10.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 -- 172.21.15.188:0/454375989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 msgr2=0x7f51600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:10.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 --2- 172.21.15.188:0/454375989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51600f66c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5150009fb0 tx=0x7f5150009b30).stop 2023-11-15T04:13:10.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 -- 172.21.15.188:0/454375989 shutdown_connections 2023-11-15T04:13:10.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 --2- 172.21.15.188:0/454375989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51600f66c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:10.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 -- 172.21.15.188:0/454375989 >> 172.21.15.188:0/454375989 conn(0x7f51600f1aa0 msgr2=0x7f51600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:10.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 -- 172.21.15.188:0/454375989 shutdown_connections 2023-11-15T04:13:10.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 -- 172.21.15.188:0/454375989 wait complete. 2023-11-15T04:13:10.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.177+0000 7f5164fa2700 1 Processor -- start 2023-11-15T04:13:10.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f5164fa2700 1 -- start start 2023-11-15T04:13:10.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f5164fa2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:10.181 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f5164fa2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51601069a0 con 0x7f51600f62a0 2023-11-15T04:13:10.181 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f515f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:10.181 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f515f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46254/0 (socket says 172.21.15.188:46254) 2023-11-15T04:13:10.181 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f515f7fe700 1 -- 172.21.15.188:0/117727694 learned_addr learned my addr 172.21.15.188:0/117727694 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:10.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.178+0000 7f515f7fe700 1 -- 172.21.15.188:0/117727694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5150005040 con 0x7f51600f62a0 2023-11-15T04:13:10.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f515f7fe700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5150009830 tx=0x7f515000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:10.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5150009320 con 0x7f51600f62a0 2023-11-15T04:13:10.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5160106ba0 con 0x7f51600f62a0 2023-11-15T04:13:10.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51500136c0 con 0x7f51600f62a0 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f515001ee90 con 0x7f51600f62a0 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.179+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5160107030 con 0x7f51600f62a0 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f515001e420 con 0x7f51600f62a0 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5160045ad0 con 0x7f51600f62a0 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f5147fff700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 0x7f51480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:10.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5150040ff0 con 0x7f51600f62a0 2023-11-15T04:13:10.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f515effd700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 0x7f51480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:10.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.180+0000 7f515effd700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 0x7f51480323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f51540096a0 tx=0x7f5154006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:10.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.185+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51500095f0 con 0x7f51600f62a0 2023-11-15T04:13:10.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.390+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5160107ee0 con 0x7f51600f62a0 2023-11-15T04:13:10.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.391+0000 7f5147fff700 1 -- 172.21.15.188:0/117727694 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f515001cb10 con 0x7f51600f62a0 2023-11-15T04:13:10.392 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:10.392 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:10.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 msgr2=0x7f51480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:10.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 0x7f51480323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f51540096a0 tx=0x7f5154006b40).stop 2023-11-15T04:13:10.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 msgr2=0x7f51601064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:10.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5150009830 tx=0x7f515000b040).stop 2023-11-15T04:13:10.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 shutdown_connections 2023-11-15T04:13:10.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f514802ff10 0x7f51480323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:10.394 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 --2- 172.21.15.188:0/117727694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51600f62a0 0x7f51601064b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:10.395 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 >> 172.21.15.188:0/117727694 conn(0x7f51600f1aa0 msgr2=0x7f51600f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:10.395 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 shutdown_connections 2023-11-15T04:13:10.395 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:10.392+0000 7f5164fa2700 1 -- 172.21.15.188:0/117727694 wait complete. 2023-11-15T04:13:10.396 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:10.086028+0000 mgr.smithi038.bnbogz (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:10.392126+0000 mon.smithi038 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.188:0/117727694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:11.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:11.780 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:12.086269+0000 mgr.smithi038.bnbogz (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:13.107 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2132738258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 msgr2=0x7f8cac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:13.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2132738258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8c94009fb0 tx=0x7f8c94009b30).stop 2023-11-15T04:13:13.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2132738258 shutdown_connections 2023-11-15T04:13:13.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2132738258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:13.108 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2132738258 >> 172.21.15.188:0/2132738258 conn(0x7f8cac0f0b70 msgr2=0x7f8cac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:13.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2132738258 shutdown_connections 2023-11-15T04:13:13.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.106+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2132738258 wait complete. 2023-11-15T04:13:13.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8cb12f9700 1 Processor -- start 2023-11-15T04:13:13.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8cb12f9700 1 -- start start 2023-11-15T04:13:13.109 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8cb12f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:13.110 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8cb12f9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac060240 con 0x7f8cac0f5410 2023-11-15T04:13:13.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8caaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:13.111 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8caaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46274/0 (socket says 172.21.15.188:46274) 2023-11-15T04:13:13.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.107+0000 7f8caaffd700 1 -- 172.21.15.188:0/2324765426 learned_addr learned my addr 172.21.15.188:0/2324765426 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:13.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8caaffd700 1 -- 172.21.15.188:0/2324765426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94005040 con 0x7f8cac0f5410 2023-11-15T04:13:13.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8caaffd700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8c94014000 tx=0x7f8c9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:13.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c940095f0 con 0x7f8cac0f5410 2023-11-15T04:13:13.112 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac060440 con 0x7f8cac0f5410 2023-11-15T04:13:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c94007e70 con 0x7f8cac0f5410 2023-11-15T04:13:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c94014ea0 con 0x7f8cac0f5410 2023-11-15T04:13:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.108+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac109ab0 con 0x7f8cac0f5410 2023-11-15T04:13:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.109+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c94025070 con 0x7f8cac0f5410 2023-11-15T04:13:13.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.109+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac045ad0 con 0x7f8cac0f5410 2023-11-15T04:13:13.114 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.109+0000 7f8ca37fe700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 0x7f8c98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:13.114 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.109+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c94040970 con 0x7f8cac0f5410 2023-11-15T04:13:13.114 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.109+0000 7f8caa7fc700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 0x7f8c98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:13.114 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.110+0000 7f8caa7fc700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 0x7f8c98032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8c9c0096a0 tx=0x7f8c9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:13.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.114+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c940079c0 con 0x7f8cac0f5410 2023-11-15T04:13:13.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.321+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cac0fae60 con 0x7f8cac0f5410 2023-11-15T04:13:13.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.321+0000 7f8ca37fe700 1 -- 172.21.15.188:0/2324765426 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c9400f300 con 0x7f8cac0f5410 2023-11-15T04:13:13.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:13.322 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 msgr2=0x7f8c98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 0x7f8c98032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8c9c0096a0 tx=0x7f8c9c006b40).stop 2023-11-15T04:13:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 msgr2=0x7f8cac109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8c94014000 tx=0x7f8c9400b040).stop 2023-11-15T04:13:13.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 shutdown_connections 2023-11-15T04:13:13.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c9802ff60 0x7f8c98032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:13.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 --2- 172.21.15.188:0/2324765426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cac0f5410 0x7f8cac109400 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:13.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 >> 172.21.15.188:0/2324765426 conn(0x7f8cac0f0b70 msgr2=0x7f8cac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:13.325 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 shutdown_connections 2023-11-15T04:13:13.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:13.323+0000 7f8cb12f9700 1 -- 172.21.15.188:0/2324765426 wait complete. 2023-11-15T04:13:13.327 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:13.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:13.322694+0000 mon.smithi038 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.188:0/2324765426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:14.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:14.718 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:14.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:14.086619+0000 mgr.smithi038.bnbogz (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:15.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 -- 172.21.15.188:0/189100788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 msgr2=0x7fa1a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:15.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/189100788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a40f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fa194009fb0 tx=0x7fa194009b30).stop 2023-11-15T04:13:15.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 -- 172.21.15.188:0/189100788 shutdown_connections 2023-11-15T04:13:15.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/189100788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a40f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:15.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 -- 172.21.15.188:0/189100788 >> 172.21.15.188:0/189100788 conn(0x7fa1a40f0b70 msgr2=0x7fa1a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:15.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 -- 172.21.15.188:0/189100788 shutdown_connections 2023-11-15T04:13:15.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.895+0000 7fa1aba4e700 1 -- 172.21.15.188:0/189100788 wait complete. 2023-11-15T04:13:15.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1aba4e700 1 Processor -- start 2023-11-15T04:13:15.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1aba4e700 1 -- start start 2023-11-15T04:13:15.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1aba4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:15.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1aba4e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4108e10 con 0x7fa1a40f5410 2023-11-15T04:13:15.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1a97ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:15.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1a97ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46284/0 (socket says 172.21.15.188:46284) 2023-11-15T04:13:15.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.896+0000 7fa1a97ea700 1 -- 172.21.15.188:0/1332125231 learned_addr learned my addr 172.21.15.188:0/1332125231 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:15.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa1a97ea700 1 -- 172.21.15.188:0/1332125231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194005040 con 0x7fa1a40f5410 2023-11-15T04:13:15.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa1a97ea700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fa194013510 tx=0x7fa19400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:15.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa194009320 con 0x7fa1a40f5410 2023-11-15T04:13:15.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a4109010 con 0x7fa1a40f5410 2023-11-15T04:13:15.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa194013f00 con 0x7fa1a40f5410 2023-11-15T04:13:15.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa194014d90 con 0x7fa1a40f5410 2023-11-15T04:13:15.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.897+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a41094a0 con 0x7fa1a40f5410 2023-11-15T04:13:15.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.898+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1940095f0 con 0x7fa1a40f5410 2023-11-15T04:13:15.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.898+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a4045ad0 con 0x7fa1a40f5410 2023-11-15T04:13:15.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.898+0000 7fa199ffb700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 0x7fa190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:15.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.898+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa194041080 con 0x7fa1a40f5410 2023-11-15T04:13:15.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.898+0000 7fa1a8fe9700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 0x7fa190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:15.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.899+0000 7fa1a8fe9700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 0x7fa190032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa1a00096a0 tx=0x7fa1a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:15.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:15.903+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa194014420 con 0x7fa1a40f5410 2023-11-15T04:13:16.113 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.111+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1a40f9d80 con 0x7fa1a40f5410 2023-11-15T04:13:16.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.112+0000 7fa199ffb700 1 -- 172.21.15.188:0/1332125231 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa194010230 con 0x7fa1a40f5410 2023-11-15T04:13:16.115 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:16.115 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:16.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 msgr2=0x7fa190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:16.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 0x7fa190032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa1a00096a0 tx=0x7fa1a0006b40).stop 2023-11-15T04:13:16.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 msgr2=0x7fa1a4108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:16.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fa194013510 tx=0x7fa19400b040).stop 2023-11-15T04:13:16.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 shutdown_connections 2023-11-15T04:13:16.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa19002ff60 0x7fa190032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:16.117 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 --2- 172.21.15.188:0/1332125231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1a40f5410 0x7fa1a4108920 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:16.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 >> 172.21.15.188:0/1332125231 conn(0x7fa1a40f0b70 msgr2=0x7fa1a40f2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:16.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 shutdown_connections 2023-11-15T04:13:16.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:16.114+0000 7fa1aba4e700 1 -- 172.21.15.188:0/1332125231 wait complete. 2023-11-15T04:13:16.118 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:16.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:16.113628+0000 mon.smithi038 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.188:0/1332125231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:17.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:17.510 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:16.086955+0000 mgr.smithi038.bnbogz (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:18.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 -- 172.21.15.188:0/2164922908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 msgr2=0x7fcc700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:18.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/2164922908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc700f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fcc64009fb0 tx=0x7fcc64009b30).stop 2023-11-15T04:13:18.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 -- 172.21.15.188:0/2164922908 shutdown_connections 2023-11-15T04:13:18.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/2164922908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc700f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:18.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 -- 172.21.15.188:0/2164922908 >> 172.21.15.188:0/2164922908 conn(0x7fcc700f1c70 msgr2=0x7fcc700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:18.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 -- 172.21.15.188:0/2164922908 shutdown_connections 2023-11-15T04:13:18.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.700+0000 7fcc76d7a700 1 -- 172.21.15.188:0/2164922908 wait complete. 2023-11-15T04:13:18.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc76d7a700 1 Processor -- start 2023-11-15T04:13:18.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc76d7a700 1 -- start start 2023-11-15T04:13:18.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc76d7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:18.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc76d7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc70106bc0 con 0x7fcc700f6510 2023-11-15T04:13:18.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc74b16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:18.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc74b16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:60174/0 (socket says 172.21.15.188:60174) 2023-11-15T04:13:18.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.701+0000 7fcc74b16700 1 -- 172.21.15.188:0/4113585749 learned_addr learned my addr 172.21.15.188:0/4113585749 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:18.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc74b16700 1 -- 172.21.15.188:0/4113585749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc64005040 con 0x7fcc700f6510 2023-11-15T04:13:18.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc74b16700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fcc64013510 tx=0x7fcc6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:18.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc64009320 con 0x7fcc700f6510 2023-11-15T04:13:18.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc70106dc0 con 0x7fcc700f6510 2023-11-15T04:13:18.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc64013f00 con 0x7fcc700f6510 2023-11-15T04:13:18.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc64014d90 con 0x7fcc700f6510 2023-11-15T04:13:18.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.702+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc70107250 con 0x7fcc700f6510 2023-11-15T04:13:18.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.703+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc64014420 con 0x7fcc700f6510 2023-11-15T04:13:18.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.703+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc70045ad0 con 0x7fcc700f6510 2023-11-15T04:13:18.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.703+0000 7fcc6d7fa700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 0x7fcc60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:18.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.703+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc64041000 con 0x7fcc700f6510 2023-11-15T04:13:18.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.703+0000 7fcc6ffff700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 0x7fcc60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:18.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.704+0000 7fcc6ffff700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 0x7fcc60032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fcc5c0096a0 tx=0x7fcc5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:18.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.708+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc640095f0 con 0x7fcc700f6510 2023-11-15T04:13:18.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.913+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc70108100 con 0x7fcc700f6510 2023-11-15T04:13:18.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.913+0000 7fcc6d7fa700 1 -- 172.21.15.188:0/4113585749 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc6401db70 con 0x7fcc700f6510 2023-11-15T04:13:18.915 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:18.915 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 msgr2=0x7fcc60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:18.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 0x7fcc60032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fcc5c0096a0 tx=0x7fcc5c006b40).stop 2023-11-15T04:13:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 msgr2=0x7fcc701066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fcc64013510 tx=0x7fcc6400b040).stop 2023-11-15T04:13:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 shutdown_connections 2023-11-15T04:13:18.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcc6002ff60 0x7fcc60032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 --2- 172.21.15.188:0/4113585749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc700f6510 0x7fcc701066d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 >> 172.21.15.188:0/4113585749 conn(0x7fcc700f1c70 msgr2=0x7fcc700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 shutdown_connections 2023-11-15T04:13:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:18.915+0000 7fcc76d7a700 1 -- 172.21.15.188:0/4113585749 wait complete. 2023-11-15T04:13:18.918 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:18.087293+0000 mgr.smithi038.bnbogz (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:18.914707+0000 mon.smithi038 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.188:0/4113585749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:20.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:20.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:21.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.575+0000 7fcaf986a700 1 -- 172.21.15.188:0/1414005170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 msgr2=0x7fcaf40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:21.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.575+0000 7fcaf986a700 1 --2- 172.21.15.188:0/1414005170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf40f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fcadc009fb0 tx=0x7fcadc009b30).stop 2023-11-15T04:13:21.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.575+0000 7fcaf986a700 1 -- 172.21.15.188:0/1414005170 shutdown_connections 2023-11-15T04:13:21.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 --2- 172.21.15.188:0/1414005170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf40f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:21.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 -- 172.21.15.188:0/1414005170 >> 172.21.15.188:0/1414005170 conn(0x7fcaf40f0b70 msgr2=0x7fcaf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:21.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 -- 172.21.15.188:0/1414005170 shutdown_connections 2023-11-15T04:13:21.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 -- 172.21.15.188:0/1414005170 wait complete. 2023-11-15T04:13:21.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 Processor -- start 2023-11-15T04:13:21.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.576+0000 7fcaf986a700 1 -- start start 2023-11-15T04:13:21.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf986a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:21.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf986a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf4060280 con 0x7fcaf40f5410 2023-11-15T04:13:21.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:21.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:60188/0 (socket says 172.21.15.188:60188) 2023-11-15T04:13:21.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf2ffd700 1 -- 172.21.15.188:0/3188622859 learned_addr learned my addr 172.21.15.188:0/3188622859 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:21.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf2ffd700 1 -- 172.21.15.188:0/3188622859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc005040 con 0x7fcaf40f5410 2023-11-15T04:13:21.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.577+0000 7fcaf2ffd700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fcadc014000 tx=0x7fcadc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcadc0095f0 con 0x7fcaf40f5410 2023-11-15T04:13:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf4060480 con 0x7fcaf40f5410 2023-11-15T04:13:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcadc007e70 con 0x7fcaf40f5410 2023-11-15T04:13:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcadc014ea0 con 0x7fcaf40f5410 2023-11-15T04:13:21.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4109af0 con 0x7fcaf40f5410 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcadc025070 con 0x7fcaf40f5410 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.578+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf4045ad0 con 0x7fcaf40f5410 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.579+0000 7fcaeb7fe700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 0x7fcae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.579+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcadc040980 con 0x7fcaf40f5410 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.579+0000 7fcaf27fc700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 0x7fcae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:21.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.579+0000 7fcaf27fc700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 0x7fcae0032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcae40096a0 tx=0x7fcae4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:21.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.583+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcadc0079c0 con 0x7fcaf40f5410 2023-11-15T04:13:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:20.087550+0000 mgr.smithi038.bnbogz (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:21.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.788+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaf410a310 con 0x7fcaf40f5410 2023-11-15T04:13:21.789 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.788+0000 7fcaeb7fe700 1 -- 172.21.15.188:0/3188622859 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcadc00f300 con 0x7fcaf40f5410 2023-11-15T04:13:21.789 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:21.789 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:21.790 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 msgr2=0x7fcae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:21.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 0x7fcae0032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcae40096a0 tx=0x7fcae4006b40).stop 2023-11-15T04:13:21.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 msgr2=0x7fcaf41093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:21.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fcadc014000 tx=0x7fcadc00b040).stop 2023-11-15T04:13:21.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 shutdown_connections 2023-11-15T04:13:21.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcae002ff60 0x7fcae0032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:21.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 --2- 172.21.15.188:0/3188622859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcaf40f5410 0x7fcaf41093c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:21.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 >> 172.21.15.188:0/3188622859 conn(0x7fcaf40f0b70 msgr2=0x7fcaf40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:21.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 shutdown_connections 2023-11-15T04:13:21.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:21.790+0000 7fcaf986a700 1 -- 172.21.15.188:0/3188622859 wait complete. 2023-11-15T04:13:21.793 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:21.789795+0000 mon.smithi038 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.188:0/3188622859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:22.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:22.087896+0000 mgr.smithi038.bnbogz (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:23.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:23.212 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:24.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- 172.21.15.188:0/2648506355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 msgr2=0x7f5d380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:24.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/2648506355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d380f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f5d28009fb0 tx=0x7f5d28009b30).stop 2023-11-15T04:13:24.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- 172.21.15.188:0/2648506355 shutdown_connections 2023-11-15T04:13:24.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/2648506355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d380f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:24.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- 172.21.15.188:0/2648506355 >> 172.21.15.188:0/2648506355 conn(0x7f5d380f09a0 msgr2=0x7f5d380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:24.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- 172.21.15.188:0/2648506355 shutdown_connections 2023-11-15T04:13:24.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- 172.21.15.188:0/2648506355 wait complete. 2023-11-15T04:13:24.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 Processor -- start 2023-11-15T04:13:24.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- start start 2023-11-15T04:13:24.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:24.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.402+0000 7f5d3e21d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d38060220 con 0x7f5d380f51a0 2023-11-15T04:13:24.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3d21b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:24.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3d21b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:60206/0 (socket says 172.21.15.188:60206) 2023-11-15T04:13:24.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3d21b700 1 -- 172.21.15.188:0/845484731 learned_addr learned my addr 172.21.15.188:0/845484731 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:24.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3d21b700 1 -- 172.21.15.188:0/845484731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d28005040 con 0x7f5d380f51a0 2023-11-15T04:13:24.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3d21b700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f5d28009830 tx=0x7f5d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:24.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d28009320 con 0x7f5d380f51a0 2023-11-15T04:13:24.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d38060420 con 0x7f5d380f51a0 2023-11-15T04:13:24.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d280136c0 con 0x7f5d380f51a0 2023-11-15T04:13:24.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.403+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d2801ee90 con 0x7f5d380f51a0 2023-11-15T04:13:24.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.404+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d38109880 con 0x7f5d380f51a0 2023-11-15T04:13:24.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.404+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d28014440 con 0x7f5d380f51a0 2023-11-15T04:13:24.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.404+0000 7f5d2dffb700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 0x7f5d240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:24.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.404+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d28040ec0 con 0x7f5d380f51a0 2023-11-15T04:13:24.410 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.404+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d38059100 con 0x7f5d380f51a0 2023-11-15T04:13:24.410 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.405+0000 7f5d3ca1a700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 0x7f5d240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:24.410 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.405+0000 7f5d3ca1a700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 0x7f5d240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5d340096a0 tx=0x7f5d34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:24.410 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.410+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d2801c910 con 0x7f5d380f51a0 2023-11-15T04:13:24.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.614+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d38045ad0 con 0x7f5d380f51a0 2023-11-15T04:13:24.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.615+0000 7f5d2dffb700 1 -- 172.21.15.188:0/845484731 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d38045ad0 con 0x7f5d380f51a0 2023-11-15T04:13:24.617 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:24.617 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:24.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.616+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 msgr2=0x7f5d240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:24.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.616+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 0x7f5d240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5d340096a0 tx=0x7f5d34006b40).stop 2023-11-15T04:13:24.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.616+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 msgr2=0x7f5d381091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:24.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.616+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f5d28009830 tx=0x7f5d2800b040).stop 2023-11-15T04:13:24.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 shutdown_connections 2023-11-15T04:13:24.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5d2402ff10 0x7f5d240323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:24.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 --2- 172.21.15.188:0/845484731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d380f51a0 0x7f5d381091d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:24.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 >> 172.21.15.188:0/845484731 conn(0x7f5d380f09a0 msgr2=0x7f5d380f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:24.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 shutdown_connections 2023-11-15T04:13:24.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:24.617+0000 7f5d3e21d700 1 -- 172.21.15.188:0/845484731 wait complete. 2023-11-15T04:13:24.620 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:24.088112+0000 mgr.smithi038.bnbogz (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:24.616314+0000 mon.smithi038 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.188:0/845484731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:26.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:26.076 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:27.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 -- 172.21.15.188:0/4008407428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 msgr2=0x7f3b640f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:27.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 --2- 172.21.15.188:0/4008407428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b640f4be0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3b58009fb0 tx=0x7f3b58009b30).stop 2023-11-15T04:13:27.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 -- 172.21.15.188:0/4008407428 shutdown_connections 2023-11-15T04:13:27.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 --2- 172.21.15.188:0/4008407428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b640f4be0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:27.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 -- 172.21.15.188:0/4008407428 >> 172.21.15.188:0/4008407428 conn(0x7f3b640f02e0 msgr2=0x7f3b640f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:27.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 -- 172.21.15.188:0/4008407428 shutdown_connections 2023-11-15T04:13:27.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.248+0000 7f3b6b187700 1 -- 172.21.15.188:0/4008407428 wait complete. 2023-11-15T04:13:27.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b6b187700 1 Processor -- start 2023-11-15T04:13:27.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b6b187700 1 -- start start 2023-11-15T04:13:27.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b6b187700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:27.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b6b187700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64106bf0 con 0x7f3b640f47c0 2023-11-15T04:13:27.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b68f23700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:27.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b68f23700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:60220/0 (socket says 172.21.15.188:60220) 2023-11-15T04:13:27.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.249+0000 7f3b68f23700 1 -- 172.21.15.188:0/1217303754 learned_addr learned my addr 172.21.15.188:0/1217303754 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:27.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b68f23700 1 -- 172.21.15.188:0/1217303754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b58005040 con 0x7f3b640f47c0 2023-11-15T04:13:27.253 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b68f23700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3b58013510 tx=0x7f3b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:27.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b58009320 con 0x7f3b640f47c0 2023-11-15T04:13:27.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64106df0 con 0x7f3b640f47c0 2023-11-15T04:13:27.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b58013f00 con 0x7f3b640f47c0 2023-11-15T04:13:27.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b58014d90 con 0x7f3b640f47c0 2023-11-15T04:13:27.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.250+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64107280 con 0x7f3b640f47c0 2023-11-15T04:13:27.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.251+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b580095f0 con 0x7f3b640f47c0 2023-11-15T04:13:27.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.251+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b64045ad0 con 0x7f3b640f47c0 2023-11-15T04:13:27.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.251+0000 7f3b617fa700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 0x7f3b54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:27.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.251+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b58040fe0 con 0x7f3b640f47c0 2023-11-15T04:13:27.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.251+0000 7f3b63fff700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 0x7f3b54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:27.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.252+0000 7f3b63fff700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 0x7f3b54032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3b500096a0 tx=0x7f3b50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:27.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.256+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b5806d2b0 con 0x7f3b640f47c0 2023-11-15T04:13:27.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.461+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b64108130 con 0x7f3b640f47c0 2023-11-15T04:13:27.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.462+0000 7f3b617fa700 1 -- 172.21.15.188:0/1217303754 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b58014420 con 0x7f3b640f47c0 2023-11-15T04:13:27.463 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:27.463 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:27.464 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 msgr2=0x7f3b54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:27.464 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 0x7f3b54032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3b500096a0 tx=0x7f3b50006b40).stop 2023-11-15T04:13:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 msgr2=0x7f3b64106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3b58013510 tx=0x7f3b5800b040).stop 2023-11-15T04:13:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 shutdown_connections 2023-11-15T04:13:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3b5402ffb0 0x7f3b54032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:27.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 --2- 172.21.15.188:0/1217303754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b640f47c0 0x7f3b64106700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:27.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.463+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 >> 172.21.15.188:0/1217303754 conn(0x7f3b640f02e0 msgr2=0x7f3b640f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:27.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.464+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 shutdown_connections 2023-11-15T04:13:27.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:27.464+0000 7f3b6b187700 1 -- 172.21.15.188:0/1217303754 wait complete. 2023-11-15T04:13:27.466 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:27.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:26.088310+0000 mgr.smithi038.bnbogz (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:28.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:27.463282+0000 mon.smithi038 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.188:0/1217303754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:28.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:28.890 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:28.088605+0000 mgr.smithi038.bnbogz (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:30.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.051+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3414400795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 msgr2=0x7fd0dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:30.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.051+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3414400795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc0f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fd0d0009fb0 tx=0x7fd0d0009b30).stop 2023-11-15T04:13:30.053 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.051+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3414400795 shutdown_connections 2023-11-15T04:13:30.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.051+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3414400795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc0f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:30.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.051+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3414400795 >> 172.21.15.188:0/3414400795 conn(0x7fd0dc0f0b70 msgr2=0x7fd0dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:30.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3414400795 shutdown_connections 2023-11-15T04:13:30.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3414400795 wait complete. 2023-11-15T04:13:30.054 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 Processor -- start 2023-11-15T04:13:30.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 -- start start 2023-11-15T04:13:30.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:30.055 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.052+0000 7fd0e3de4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc060240 con 0x7fd0dc0f5410 2023-11-15T04:13:30.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e1b80700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:30.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e1b80700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:53262/0 (socket says 172.21.15.188:53262) 2023-11-15T04:13:30.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e1b80700 1 -- 172.21.15.188:0/3594670752 learned_addr learned my addr 172.21.15.188:0/3594670752 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:30.057 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e1b80700 1 -- 172.21.15.188:0/3594670752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0005040 con 0x7fd0dc0f5410 2023-11-15T04:13:30.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e1b80700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd0d0013510 tx=0x7fd0d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:30.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d0009320 con 0x7fd0dc0f5410 2023-11-15T04:13:30.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc060440 con 0x7fd0dc0f5410 2023-11-15T04:13:30.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0d0013f00 con 0x7fd0dc0f5410 2023-11-15T04:13:30.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d0014d90 con 0x7fd0dc0f5410 2023-11-15T04:13:30.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.053+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc109ae0 con 0x7fd0dc0f5410 2023-11-15T04:13:30.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.054+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0d00095f0 con 0x7fd0dc0f5410 2023-11-15T04:13:30.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.054+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc045ad0 con 0x7fd0dc0f5410 2023-11-15T04:13:30.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.054+0000 7fd0ce7fc700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 0x7fd0c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:30.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.054+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0d00410a0 con 0x7fd0dc0f5410 2023-11-15T04:13:30.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.055+0000 7fd0e137f700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 0x7fd0c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:30.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.055+0000 7fd0e137f700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 0x7fd0c8032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fd0d80096a0 tx=0x7fd0d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:30.061 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.059+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0d001d7f0 con 0x7fd0dc0f5410 2023-11-15T04:13:30.270 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.269+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0dc0fae60 con 0x7fd0dc0f5410 2023-11-15T04:13:30.271 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.270+0000 7fd0ce7fc700 1 -- 172.21.15.188:0/3594670752 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0d0010230 con 0x7fd0dc0f5410 2023-11-15T04:13:30.271 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:30.271 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:30.272 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 msgr2=0x7fd0c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:30.272 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 0x7fd0c8032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fd0d80096a0 tx=0x7fd0d8006b40).stop 2023-11-15T04:13:30.273 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 msgr2=0x7fd0dc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:30.273 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fd0d0013510 tx=0x7fd0d000b040).stop 2023-11-15T04:13:30.273 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 shutdown_connections 2023-11-15T04:13:30.273 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0c802ff60 0x7fd0c8032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:30.273 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 --2- 172.21.15.188:0/3594670752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0dc0f5410 0x7fd0dc109430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 >> 172.21.15.188:0/3594670752 conn(0x7fd0dc0f0b70 msgr2=0x7fd0dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 shutdown_connections 2023-11-15T04:13:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:30.271+0000 7fd0e3de4700 1 -- 172.21.15.188:0/3594670752 wait complete. 2023-11-15T04:13:30.274 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:30.088983+0000 mgr.smithi038.bnbogz (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:30.271036+0000 mon.smithi038 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.188:0/3594670752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:31.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:31.667 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:32.089303+0000 mgr.smithi038.bnbogz (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:32.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 -- 172.21.15.188:0/100702305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 msgr2=0x7f12280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:32.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 --2- 172.21.15.188:0/100702305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12280f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1218009fb0 tx=0x7f1218009b30).stop 2023-11-15T04:13:32.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 -- 172.21.15.188:0/100702305 shutdown_connections 2023-11-15T04:13:32.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 --2- 172.21.15.188:0/100702305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12280f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:32.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 -- 172.21.15.188:0/100702305 >> 172.21.15.188:0/100702305 conn(0x7f12280f09a0 msgr2=0x7f12280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:32.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 -- 172.21.15.188:0/100702305 shutdown_connections 2023-11-15T04:13:32.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 -- 172.21.15.188:0/100702305 wait complete. 2023-11-15T04:13:32.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.909+0000 7f1227fff700 1 Processor -- start 2023-11-15T04:13:32.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1227fff700 1 -- start start 2023-11-15T04:13:32.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1227fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:32.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1227fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12281069a0 con 0x7f12280f51a0 2023-11-15T04:13:32.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1226ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:32.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1226ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:53280/0 (socket says 172.21.15.188:53280) 2023-11-15T04:13:32.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1226ffd700 1 -- 172.21.15.188:0/2371029964 learned_addr learned my addr 172.21.15.188:0/2371029964 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:32.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1226ffd700 1 -- 172.21.15.188:0/2371029964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218005040 con 0x7f12280f51a0 2023-11-15T04:13:32.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.910+0000 7f1226ffd700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1218013510 tx=0x7f121800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:32.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.911+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1218009320 con 0x7f12280f51a0 2023-11-15T04:13:32.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.911+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1228106ba0 con 0x7f12280f51a0 2023-11-15T04:13:32.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.911+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1228107030 con 0x7f12280f51a0 2023-11-15T04:13:32.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.911+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1218013f00 con 0x7f12280f51a0 2023-11-15T04:13:32.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.911+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1218014cc0 con 0x7f12280f51a0 2023-11-15T04:13:32.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1218014420 con 0x7f12280f51a0 2023-11-15T04:13:32.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1208004fa0 con 0x7f12280f51a0 2023-11-15T04:13:32.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f120f7fe700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 0x7f1210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:32.916 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1218041130 con 0x7f12280f51a0 2023-11-15T04:13:32.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f12267fc700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 0x7f1210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:32.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.912+0000 7f12267fc700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 0x7f1210032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f121c0096a0 tx=0x7f121c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:32.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:32.917+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12180095f0 con 0x7f12280f51a0 2023-11-15T04:13:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.122+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1208005670 con 0x7f12280f51a0 2023-11-15T04:13:33.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.123+0000 7f120f7fe700 1 -- 172.21.15.188:0/2371029964 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12180095f0 con 0x7f12280f51a0 2023-11-15T04:13:33.124 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:33.124 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:33.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 msgr2=0x7f1210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:33.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 0x7f1210032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f121c0096a0 tx=0x7f121c006b40).stop 2023-11-15T04:13:33.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 msgr2=0x7f12281064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:33.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1218013510 tx=0x7f121800b040).stop 2023-11-15T04:13:33.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 shutdown_connections 2023-11-15T04:13:33.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f121002ff60 0x7f1210032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:33.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 --2- 172.21.15.188:0/2371029964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12280f51a0 0x7f12281064b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:33.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 >> 172.21.15.188:0/2371029964 conn(0x7f12280f09a0 msgr2=0x7f12280f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:33.127 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 shutdown_connections 2023-11-15T04:13:33.127 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:33.124+0000 7f1227fff700 1 -- 172.21.15.188:0/2371029964 wait complete. 2023-11-15T04:13:33.127 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:33.124166+0000 mon.smithi038 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.188:0/2371029964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:34.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:34.524 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:34.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:34.089599+0000 mgr.smithi038.bnbogz (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:35.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/206914850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 msgr2=0x7f4abc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:35.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/206914850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f4ab0009fb0 tx=0x7f4ab0009b30).stop 2023-11-15T04:13:35.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/206914850 shutdown_connections 2023-11-15T04:13:35.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/206914850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/206914850 >> 172.21.15.188:0/206914850 conn(0x7f4abc0f0b70 msgr2=0x7f4abc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/206914850 shutdown_connections 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.705+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/206914850 wait complete. 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4ac1dbd700 1 Processor -- start 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4ac1dbd700 1 -- start start 2023-11-15T04:13:35.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4ac1dbd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:35.708 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4ac1dbd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc105b20 con 0x7f4abc0f5410 2023-11-15T04:13:35.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4abb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:35.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4abb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:53298/0 (socket says 172.21.15.188:53298) 2023-11-15T04:13:35.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.706+0000 7f4abb7fe700 1 -- 172.21.15.188:0/2984951447 learned_addr learned my addr 172.21.15.188:0/2984951447 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:35.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4abb7fe700 1 -- 172.21.15.188:0/2984951447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab0005040 con 0x7f4abc0f5410 2023-11-15T04:13:35.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4abb7fe700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f4ab0013510 tx=0x7f4ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:35.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ab0009320 con 0x7f4abc0f5410 2023-11-15T04:13:35.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc105d20 con 0x7f4abc0f5410 2023-11-15T04:13:35.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ab0013f00 con 0x7f4abc0f5410 2023-11-15T04:13:35.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ab0014d90 con 0x7f4abc0f5410 2023-11-15T04:13:35.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.707+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc1061b0 con 0x7f4abc0f5410 2023-11-15T04:13:35.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.708+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ab00095f0 con 0x7f4abc0f5410 2023-11-15T04:13:35.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.708+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc045ad0 con 0x7f4abc0f5410 2023-11-15T04:13:35.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.708+0000 7f4aa3fff700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:35.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.708+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ab0041080 con 0x7f4abc0f5410 2023-11-15T04:13:35.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.708+0000 7f4abaffd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:35.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.709+0000 7f4abaffd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 0x7f4aa4032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4aac0096a0 tx=0x7f4aac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:35.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.713+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ab006d2b0 con 0x7f4abc0f5410 2023-11-15T04:13:35.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.920+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4abc0fae60 con 0x7f4abc0f5410 2023-11-15T04:13:35.922 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.921+0000 7f4aa3fff700 1 -- 172.21.15.188:0/2984951447 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ab0014420 con 0x7f4abc0f5410 2023-11-15T04:13:35.922 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:35.922 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:35.923 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 msgr2=0x7f4aa4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:35.923 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 0x7f4aa4032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4aac0096a0 tx=0x7f4aac006b40).stop 2023-11-15T04:13:35.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 msgr2=0x7f4abc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:35.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f4ab0013510 tx=0x7f4ab000b040).stop 2023-11-15T04:13:35.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 shutdown_connections 2023-11-15T04:13:35.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:35.924 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 --2- 172.21.15.188:0/2984951447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4abc0f5410 0x7f4abc105630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:35.925 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 >> 172.21.15.188:0/2984951447 conn(0x7f4abc0f0b70 msgr2=0x7f4abc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:35.925 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 shutdown_connections 2023-11-15T04:13:35.925 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:35.923+0000 7f4ac1dbd700 1 -- 172.21.15.188:0/2984951447 wait complete. 2023-11-15T04:13:35.925 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:36.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:35.921364+0000 mon.smithi038 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.188:0/2984951447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:37.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:37.324 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:37.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:36 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:36.089946+0000 mgr.smithi038.bnbogz (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:38.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.507+0000 7f249a7fc700 1 -- 172.21.15.188:0/1358397045 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f248c0138a0 con 0x7f249c0f51a0 2023-11-15T04:13:38.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 -- 172.21.15.188:0/1358397045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 msgr2=0x7f249c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:38.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/1358397045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c0f55c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f248c009fb0 tx=0x7f248c009b30).stop 2023-11-15T04:13:38.508 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 -- 172.21.15.188:0/1358397045 shutdown_connections 2023-11-15T04:13:38.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/1358397045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c0f55c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:38.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 -- 172.21.15.188:0/1358397045 >> 172.21.15.188:0/1358397045 conn(0x7f249c0f09a0 msgr2=0x7f249c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:38.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 -- 172.21.15.188:0/1358397045 shutdown_connections 2023-11-15T04:13:38.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 -- 172.21.15.188:0/1358397045 wait complete. 2023-11-15T04:13:38.509 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.508+0000 7f24a0ee8700 1 Processor -- start 2023-11-15T04:13:38.511 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f24a0ee8700 1 -- start start 2023-11-15T04:13:38.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f24a0ee8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:38.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f24a0ee8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c108bc0 con 0x7f249c0f51a0 2023-11-15T04:13:38.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f249b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:38.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f249b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59612/0 (socket says 172.21.15.188:59612) 2023-11-15T04:13:38.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f249b7fe700 1 -- 172.21.15.188:0/3268311669 learned_addr learned my addr 172.21.15.188:0/3268311669 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.509+0000 7f249b7fe700 1 -- 172.21.15.188:0/3268311669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c005040 con 0x7f249c0f51a0 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f249b7fe700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f248c013510 tx=0x7f248c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f248c009320 con 0x7f249c0f51a0 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c108dc0 con 0x7f249c0f51a0 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f248c013f00 con 0x7f249c0f51a0 2023-11-15T04:13:38.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f248c014dc0 con 0x7f249c0f51a0 2023-11-15T04:13:38.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c109250 con 0x7f249c0f51a0 2023-11-15T04:13:38.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.510+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f248c014420 con 0x7f249c0f51a0 2023-11-15T04:13:38.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.511+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c045ad0 con 0x7f249c0f51a0 2023-11-15T04:13:38.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.511+0000 7f2483fff700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 0x7f2484032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:38.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.511+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f248c041050 con 0x7f249c0f51a0 2023-11-15T04:13:38.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.511+0000 7f249affd700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 0x7f2484032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:38.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.511+0000 7f249affd700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 0x7f2484032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f24900096a0 tx=0x7f2490006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:38.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.516+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f248c0095f0 con 0x7f249c0f51a0 2023-11-15T04:13:38.720 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.720+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f249c0f9b20 con 0x7f249c0f51a0 2023-11-15T04:13:38.720 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.721+0000 7f2483fff700 1 -- 172.21.15.188:0/3268311669 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f248c01cb10 con 0x7f249c0f51a0 2023-11-15T04:13:38.721 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:38.721 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:38.722 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 msgr2=0x7f2484032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:38.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 0x7f2484032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f24900096a0 tx=0x7f2490006b40).stop 2023-11-15T04:13:38.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 msgr2=0x7f249c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:38.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f248c013510 tx=0x7f248c00b040).stop 2023-11-15T04:13:38.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 shutdown_connections 2023-11-15T04:13:38.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f248402ff60 0x7f2484032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:38.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 --2- 172.21.15.188:0/3268311669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f249c0f51a0 0x7f249c1086d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:38.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 >> 172.21.15.188:0/3268311669 conn(0x7f249c0f09a0 msgr2=0x7f249c0f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:38.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 shutdown_connections 2023-11-15T04:13:38.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:38.722+0000 7f24a0ee8700 1 -- 172.21.15.188:0/3268311669 wait complete. 2023-11-15T04:13:38.725 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:38.090127+0000 mgr.smithi038.bnbogz (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:38.721065+0000 mon.smithi038 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.188:0/3268311669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:40.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:40.141 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:41.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.308+0000 7fd0d9c30700 1 -- 172.21.15.188:0/1699496267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 msgr2=0x7fd0d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:41.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.308+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/1699496267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d40f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fd0c4009fb0 tx=0x7fd0c4009b30).stop 2023-11-15T04:13:41.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.308+0000 7fd0d9c30700 1 -- 172.21.15.188:0/1699496267 shutdown_connections 2023-11-15T04:13:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.308+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/1699496267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d40f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.308+0000 7fd0d9c30700 1 -- 172.21.15.188:0/1699496267 >> 172.21.15.188:0/1699496267 conn(0x7fd0d40f0b70 msgr2=0x7fd0d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.309+0000 7fd0d9c30700 1 -- 172.21.15.188:0/1699496267 shutdown_connections 2023-11-15T04:13:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.309+0000 7fd0d9c30700 1 -- 172.21.15.188:0/1699496267 wait complete. 2023-11-15T04:13:41.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.309+0000 7fd0d9c30700 1 Processor -- start 2023-11-15T04:13:41.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.309+0000 7fd0d9c30700 1 -- start start 2023-11-15T04:13:41.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d9c30700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:41.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d9c30700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4106c00 con 0x7fd0d40f5410 2023-11-15T04:13:41.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:41.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59624/0 (socket says 172.21.15.188:59624) 2023-11-15T04:13:41.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d37fe700 1 -- 172.21.15.188:0/180673541 learned_addr learned my addr 172.21.15.188:0/180673541 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:41.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d37fe700 1 -- 172.21.15.188:0/180673541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c4005040 con 0x7fd0d40f5410 2023-11-15T04:13:41.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0d37fe700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd0c4013510 tx=0x7fd0c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:41.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.310+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c4009320 con 0x7fd0d40f5410 2023-11-15T04:13:41.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4106e00 con 0x7fd0d40f5410 2023-11-15T04:13:41.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0c4013f00 con 0x7fd0d40f5410 2023-11-15T04:13:41.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0c4014d90 con 0x7fd0d40f5410 2023-11-15T04:13:41.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4107290 con 0x7fd0d40f5410 2023-11-15T04:13:41.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0c40263a0 con 0x7fd0d40f5410 2023-11-15T04:13:41.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0bbfff700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 0x7fd0bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:41.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.311+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d4045ad0 con 0x7fd0d40f5410 2023-11-15T04:13:41.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.312+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0c4041300 con 0x7fd0d40f5410 2023-11-15T04:13:41.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.312+0000 7fd0d2ffd700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 0x7fd0bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:41.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.312+0000 7fd0d2ffd700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 0x7fd0bc032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd0c80096a0 tx=0x7fd0c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:41.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.316+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0c401ccf0 con 0x7fd0d40f5410 2023-11-15T04:13:41.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.522+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0d4108140 con 0x7fd0d40f5410 2023-11-15T04:13:41.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.523+0000 7fd0bbfff700 1 -- 172.21.15.188:0/180673541 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0c401cb10 con 0x7fd0d40f5410 2023-11-15T04:13:41.523 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:41.523 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:41.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 msgr2=0x7fd0bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:41.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 0x7fd0bc032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd0c80096a0 tx=0x7fd0c8006b40).stop 2023-11-15T04:13:41.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 msgr2=0x7fd0d4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:41.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd0c4013510 tx=0x7fd0c400b040).stop 2023-11-15T04:13:41.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 shutdown_connections 2023-11-15T04:13:41.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd0bc02ff60 0x7fd0bc032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:41.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 --2- 172.21.15.188:0/180673541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d40f5410 0x7fd0d4106710 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:41.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 >> 172.21.15.188:0/180673541 conn(0x7fd0d40f0b70 msgr2=0x7fd0d405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:41.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 shutdown_connections 2023-11-15T04:13:41.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:41.524+0000 7fd0d9c30700 1 -- 172.21.15.188:0/180673541 wait complete. 2023-11-15T04:13:41.526 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:40.090340+0000 mgr.smithi038.bnbogz (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:41.522961+0000 mon.smithi038 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.188:0/180673541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:42.090648+0000 mgr.smithi038.bnbogz (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:42.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:42.913 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:44.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.154+0000 7f4b57e1f700 1 -- 172.21.15.188:0/2485096035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 msgr2=0x7f4b500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:44.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.154+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/2485096035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b500f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f4b40009fb0 tx=0x7f4b40009b30).stop 2023-11-15T04:13:44.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- 172.21.15.188:0/2485096035 shutdown_connections 2023-11-15T04:13:44.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/2485096035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b500f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:44.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- 172.21.15.188:0/2485096035 >> 172.21.15.188:0/2485096035 conn(0x7f4b500f0b70 msgr2=0x7f4b500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:44.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- 172.21.15.188:0/2485096035 shutdown_connections 2023-11-15T04:13:44.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- 172.21.15.188:0/2485096035 wait complete. 2023-11-15T04:13:44.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 Processor -- start 2023-11-15T04:13:44.157 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- start start 2023-11-15T04:13:44.157 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:44.157 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.155+0000 7f4b57e1f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50060280 con 0x7f4b500f5410 2023-11-15T04:13:44.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b55bbb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:44.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b55bbb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59648/0 (socket says 172.21.15.188:59648) 2023-11-15T04:13:44.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b55bbb700 1 -- 172.21.15.188:0/4176766810 learned_addr learned my addr 172.21.15.188:0/4176766810 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:44.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b55bbb700 1 -- 172.21.15.188:0/4176766810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40005040 con 0x7f4b500f5410 2023-11-15T04:13:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b55bbb700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4b40013510 tx=0x7f4b4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b40009320 con 0x7f4b500f5410 2023-11-15T04:13:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50060480 con 0x7f4b500f5410 2023-11-15T04:13:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b40013f00 con 0x7f4b500f5410 2023-11-15T04:13:44.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b40014d90 con 0x7f4b500f5410 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.156+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50109b20 con 0x7f4b500f5410 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.157+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b400095f0 con 0x7f4b500f5410 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.157+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50045ad0 con 0x7f4b500f5410 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.157+0000 7f4b467fc700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.157+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b400410c0 con 0x7f4b500f5410 2023-11-15T04:13:44.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.157+0000 7f4b553ba700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:44.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.158+0000 7f4b553ba700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 0x7f4b3c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4b4c0096a0 tx=0x7f4b4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:44.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.162+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b40014420 con 0x7f4b500f5410 2023-11-15T04:13:44.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.366+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b5010a340 con 0x7f4b500f5410 2023-11-15T04:13:44.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.366+0000 7f4b467fc700 1 -- 172.21.15.188:0/4176766810 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b40010230 con 0x7f4b500f5410 2023-11-15T04:13:44.366 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:44.367 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:44.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 msgr2=0x7f4b3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:44.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 0x7f4b3c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4b4c0096a0 tx=0x7f4b4c006b40).stop 2023-11-15T04:13:44.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 msgr2=0x7f4b501093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4b40013510 tx=0x7f4b4000b040).stop 2023-11-15T04:13:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 shutdown_connections 2023-11-15T04:13:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 --2- 172.21.15.188:0/4176766810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b500f5410 0x7f4b501093f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:44.369 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 >> 172.21.15.188:0/4176766810 conn(0x7f4b500f0b70 msgr2=0x7f4b500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:44.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 shutdown_connections 2023-11-15T04:13:44.370 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:44.368+0000 7f4b57e1f700 1 -- 172.21.15.188:0/4176766810 wait complete. 2023-11-15T04:13:44.370 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:44.090907+0000 mgr.smithi038.bnbogz (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:45.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:44.366821+0000 mon.smithi038 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.188:0/4176766810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:45.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:45.758 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:46.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 -- 172.21.15.188:0/2260315910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 msgr2=0x7efe040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:46.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 --2- 172.21.15.188:0/2260315910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe040f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7efdf4009fb0 tx=0x7efdf4009b30).stop 2023-11-15T04:13:46.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 -- 172.21.15.188:0/2260315910 shutdown_connections 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 --2- 172.21.15.188:0/2260315910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe040f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 -- 172.21.15.188:0/2260315910 >> 172.21.15.188:0/2260315910 conn(0x7efe040f1c70 msgr2=0x7efe040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 -- 172.21.15.188:0/2260315910 shutdown_connections 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.973+0000 7efe0b490700 1 -- 172.21.15.188:0/2260315910 wait complete. 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0b490700 1 Processor -- start 2023-11-15T04:13:46.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0b490700 1 -- start start 2023-11-15T04:13:46.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0b490700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:46.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0b490700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04106c20 con 0x7efe040f6510 2023-11-15T04:13:46.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0922c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:46.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0922c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59674/0 (socket says 172.21.15.188:59674) 2023-11-15T04:13:46.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.974+0000 7efe0922c700 1 -- 172.21.15.188:0/1990500632 learned_addr learned my addr 172.21.15.188:0/1990500632 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:46.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efe0922c700 1 -- 172.21.15.188:0/1990500632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4005040 con 0x7efe040f6510 2023-11-15T04:13:46.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efe0922c700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efdf4013510 tx=0x7efdf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:46.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf4009320 con 0x7efe040f6510 2023-11-15T04:13:46.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe04106e20 con 0x7efe040f6510 2023-11-15T04:13:46.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdf4013f00 con 0x7efe040f6510 2023-11-15T04:13:46.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf4014d90 con 0x7efe040f6510 2023-11-15T04:13:46.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.975+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe041072b0 con 0x7efe040f6510 2023-11-15T04:13:46.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.976+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdf4014420 con 0x7efe040f6510 2023-11-15T04:13:46.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.976+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe04045ad0 con 0x7efe040f6510 2023-11-15T04:13:46.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.976+0000 7efdf9ffb700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 0x7efdf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:46.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.976+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdf40410c0 con 0x7efe040f6510 2023-11-15T04:13:46.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.976+0000 7efe08a2b700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 0x7efdf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:46.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.977+0000 7efe08a2b700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 0x7efdf0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7efe000096a0 tx=0x7efe00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:46.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:46.981+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdf40095f0 con 0x7efe040f6510 2023-11-15T04:13:47.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.185+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe040fbf60 con 0x7efe040f6510 2023-11-15T04:13:47.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.186+0000 7efdf9ffb700 1 -- 172.21.15.188:0/1990500632 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdf401db70 con 0x7efe040f6510 2023-11-15T04:13:47.187 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:47.188 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:47.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 msgr2=0x7efdf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:47.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 0x7efdf0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7efe000096a0 tx=0x7efe00006b40).stop 2023-11-15T04:13:47.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 msgr2=0x7efe04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:47.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efdf4013510 tx=0x7efdf400b040).stop 2023-11-15T04:13:47.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 shutdown_connections 2023-11-15T04:13:47.190 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efdf002ff60 0x7efdf0032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:47.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 --2- 172.21.15.188:0/1990500632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe040f6510 0x7efe04106730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:47.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 >> 172.21.15.188:0/1990500632 conn(0x7efe040f1c70 msgr2=0x7efe040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:47.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 shutdown_connections 2023-11-15T04:13:47.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:47.188+0000 7efe0b490700 1 -- 172.21.15.188:0/1990500632 wait complete. 2023-11-15T04:13:47.193 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:46.091217+0000 mgr.smithi038.bnbogz (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:48.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:48.587 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:48.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:48 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:47.187451+0000 mon.smithi038 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.188:0/1990500632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:48.091424+0000 mgr.smithi038.bnbogz (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:49.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- 172.21.15.188:0/1627526068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 msgr2=0x7f10000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:49.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1627526068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10000f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f0ff0009fb0 tx=0x7f0ff0009b30).stop 2023-11-15T04:13:49.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- 172.21.15.188:0/1627526068 shutdown_connections 2023-11-15T04:13:49.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1627526068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10000f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:49.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- 172.21.15.188:0/1627526068 >> 172.21.15.188:0/1627526068 conn(0x7f10000f09a0 msgr2=0x7f10000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:49.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- 172.21.15.188:0/1627526068 shutdown_connections 2023-11-15T04:13:49.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- 172.21.15.188:0/1627526068 wait complete. 2023-11-15T04:13:49.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 Processor -- start 2023-11-15T04:13:49.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- start start 2023-11-15T04:13:49.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:49.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.771+0000 7f0fff59e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10001058a0 con 0x7f10000f51a0 2023-11-15T04:13:49.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0ffe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:49.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0ffe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45248/0 (socket says 172.21.15.188:45248) 2023-11-15T04:13:49.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0ffe59c700 1 -- 172.21.15.188:0/1032391853 learned_addr learned my addr 172.21.15.188:0/1032391853 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:49.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0ffe59c700 1 -- 172.21.15.188:0/1032391853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0005040 con 0x7f10000f51a0 2023-11-15T04:13:49.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0ffe59c700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f0ff0013510 tx=0x7f0ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:49.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ff0009320 con 0x7f10000f51a0 2023-11-15T04:13:49.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1000105aa0 con 0x7f10000f51a0 2023-11-15T04:13:49.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ff0013f00 con 0x7f10000f51a0 2023-11-15T04:13:49.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ff0014dc0 con 0x7f10000f51a0 2023-11-15T04:13:49.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.772+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1000105f30 con 0x7f10000f51a0 2023-11-15T04:13:49.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.773+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ff001d3d0 con 0x7f10000f51a0 2023-11-15T04:13:49.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.773+0000 7f0feeffd700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:49.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.773+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1000045ad0 con 0x7f10000f51a0 2023-11-15T04:13:49.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.773+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ff00410c0 con 0x7f10000f51a0 2023-11-15T04:13:49.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.774+0000 7f0ffdd9b700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:49.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.774+0000 7f0ffdd9b700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 0x7f0fe8032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0ff40096a0 tx=0x7f0ff4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:49.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.778+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ff006d2b0 con 0x7f10000f51a0 2023-11-15T04:13:49.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.984+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1000106d80 con 0x7f10000f51a0 2023-11-15T04:13:49.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.984+0000 7f0feeffd700 1 -- 172.21.15.188:0/1032391853 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ff0012510 con 0x7f10000f51a0 2023-11-15T04:13:49.985 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:49.985 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:49.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 msgr2=0x7f0fe8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:49.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 0x7f0fe8032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0ff40096a0 tx=0x7f0ff4006b40).stop 2023-11-15T04:13:49.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 msgr2=0x7f10001053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:49.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f0ff0013510 tx=0x7f0ff000b040).stop 2023-11-15T04:13:49.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 shutdown_connections 2023-11-15T04:13:49.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:49.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 --2- 172.21.15.188:0/1032391853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10000f51a0 0x7f10001053b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:49.989 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 >> 172.21.15.188:0/1032391853 conn(0x7f10000f09a0 msgr2=0x7f10000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:49.989 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 shutdown_connections 2023-11-15T04:13:49.989 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:49.986+0000 7f0fff59e700 1 -- 172.21.15.188:0/1032391853 wait complete. 2023-11-15T04:13:49.989 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:49.985811+0000 mon.smithi038 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.188:0/1032391853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:51.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:51.418 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:50.091753+0000 mgr.smithi038.bnbogz (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:50.211714+0000 mon.smithi038 (mon.0) 221 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:13:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:50.213854+0000 mon.smithi038 (mon.0) 222 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:13:52.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- 172.21.15.188:0/2230195282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 msgr2=0x7f6cb80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:52.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 --2- 172.21.15.188:0/2230195282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f6cb4009fb0 tx=0x7f6cb4009b30).stop 2023-11-15T04:13:52.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- 172.21.15.188:0/2230195282 shutdown_connections 2023-11-15T04:13:52.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 --2- 172.21.15.188:0/2230195282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:52.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- 172.21.15.188:0/2230195282 >> 172.21.15.188:0/2230195282 conn(0x7f6cb80f0b70 msgr2=0x7f6cb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:52.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- 172.21.15.188:0/2230195282 shutdown_connections 2023-11-15T04:13:52.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- 172.21.15.188:0/2230195282 wait complete. 2023-11-15T04:13:52.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 Processor -- start 2023-11-15T04:13:52.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.665+0000 7f6cbf433700 1 -- start start 2023-11-15T04:13:52.669 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbf433700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:52.669 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbf433700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80f4ff0 con 0x7f6cb8060280 2023-11-15T04:13:52.669 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbd1cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:52.669 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbd1cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45256/0 (socket says 172.21.15.188:45256) 2023-11-15T04:13:52.670 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbd1cf700 1 -- 172.21.15.188:0/462265644 learned_addr learned my addr 172.21.15.188:0/462265644 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:52.670 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbd1cf700 1 -- 172.21.15.188:0/462265644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb4005040 con 0x7f6cb8060280 2023-11-15T04:13:52.671 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.666+0000 7f6cbd1cf700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6cb4013510 tx=0x7f6cb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:52.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.667+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cb4009320 con 0x7f6cb8060280 2023-11-15T04:13:52.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.667+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb80f5190 con 0x7f6cb8060280 2023-11-15T04:13:52.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.667+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cb4013f00 con 0x7f6cb8060280 2023-11-15T04:13:52.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.667+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cb4014dc0 con 0x7f6cb8060280 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.667+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb80f5620 con 0x7f6cb8060280 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cb401d3d0 con 0x7f6cb8060280 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb8059100 con 0x7f6cb8060280 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cadffb700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 0x7f6ca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cb4041300 con 0x7f6cb8060280 2023-11-15T04:13:52.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cbc9ce700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 0x7f6ca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:52.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.668+0000 7f6cbc9ce700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 0x7f6ca4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6ca80096a0 tx=0x7f6ca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:52.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.673+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cb401ccb0 con 0x7f6cb8060280 2023-11-15T04:13:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:52.092130+0000 mgr.smithi038.bnbogz (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:52.876 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.876+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cb80fbf50 con 0x7f6cb8060280 2023-11-15T04:13:52.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.877+0000 7f6cadffb700 1 -- 172.21.15.188:0/462265644 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cb401cad0 con 0x7f6cb8060280 2023-11-15T04:13:52.877 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:52.878 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:52.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 msgr2=0x7f6ca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:52.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 0x7f6ca4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6ca80096a0 tx=0x7f6ca8006b40).stop 2023-11-15T04:13:52.879 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 msgr2=0x7f6cb80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:52.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f6cb4013510 tx=0x7f6cb400b040).stop 2023-11-15T04:13:52.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 shutdown_connections 2023-11-15T04:13:52.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6ca402ff60 0x7f6ca4032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:52.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 --2- 172.21.15.188:0/462265644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cb8060280 0x7f6cb80f6b70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:52.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 >> 172.21.15.188:0/462265644 conn(0x7f6cb80f0b70 msgr2=0x7f6cb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:52.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 shutdown_connections 2023-11-15T04:13:52.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:52.878+0000 7f6cbf433700 1 -- 172.21.15.188:0/462265644 wait complete. 2023-11-15T04:13:52.882 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:53.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:52.877884+0000 mon.smithi038 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.188:0/462265644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:54.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:54.293 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:54.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:54.092395+0000 mgr.smithi038.bnbogz (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:55.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/1341757449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 msgr2=0x7f1e040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:55.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/1341757449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e040f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1df8009fb0 tx=0x7f1df8009b30).stop 2023-11-15T04:13:55.477 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/1341757449 shutdown_connections 2023-11-15T04:13:55.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/1341757449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e040f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:55.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/1341757449 >> 172.21.15.188:0/1341757449 conn(0x7f1e040f0b70 msgr2=0x7f1e040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:55.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/1341757449 shutdown_connections 2023-11-15T04:13:55.478 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.474+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/1341757449 wait complete. 2023-11-15T04:13:55.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0b5cf700 1 Processor -- start 2023-11-15T04:13:55.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0b5cf700 1 -- start start 2023-11-15T04:13:55.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0b5cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:55.479 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0b5cf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04105b20 con 0x7f1e040f5410 2023-11-15T04:13:55.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0936b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:55.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0936b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:45274/0 (socket says 172.21.15.188:45274) 2023-11-15T04:13:55.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0936b700 1 -- 172.21.15.188:0/114368891 learned_addr learned my addr 172.21.15.188:0/114368891 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:55.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0936b700 1 -- 172.21.15.188:0/114368891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8005040 con 0x7f1e040f5410 2023-11-15T04:13:55.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.475+0000 7f1e0936b700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1df8013510 tx=0x7f1df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:55.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.476+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8009320 con 0x7f1e040f5410 2023-11-15T04:13:55.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.476+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04105d20 con 0x7f1e040f5410 2023-11-15T04:13:55.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.476+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1df8013f00 con 0x7f1e040f5410 2023-11-15T04:13:55.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.476+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8014d90 con 0x7f1e040f5410 2023-11-15T04:13:55.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.476+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e041061b0 con 0x7f1e040f5410 2023-11-15T04:13:55.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.477+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1df80095f0 con 0x7f1e040f5410 2023-11-15T04:13:55.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.477+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e04045ad0 con 0x7f1e040f5410 2023-11-15T04:13:55.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.477+0000 7f1df5ffb700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:55.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.477+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1df8041080 con 0x7f1e040f5410 2023-11-15T04:13:55.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.477+0000 7f1e08b6a700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:55.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.478+0000 7f1e08b6a700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 0x7f1df0032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1e000096a0 tx=0x7f1e00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:55.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.482+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1df8014420 con 0x7f1e040f5410 2023-11-15T04:13:55.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.688+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e040fae60 con 0x7f1e040f5410 2023-11-15T04:13:55.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.689+0000 7f1df5ffb700 1 -- 172.21.15.188:0/114368891 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1df8010230 con 0x7f1e040f5410 2023-11-15T04:13:55.690 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:55.690 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:55.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 msgr2=0x7f1df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:55.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 0x7f1df0032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1e000096a0 tx=0x7f1e00006b40).stop 2023-11-15T04:13:55.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 msgr2=0x7f1e04105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:55.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1df8013510 tx=0x7f1df800b040).stop 2023-11-15T04:13:55.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 shutdown_connections 2023-11-15T04:13:55.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1df002ff60 0x7f1df0032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:55.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 --2- 172.21.15.188:0/114368891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e040f5410 0x7f1e04105630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:55.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 >> 172.21.15.188:0/114368891 conn(0x7f1e040f0b70 msgr2=0x7f1e040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:55.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 shutdown_connections 2023-11-15T04:13:55.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:55.691+0000 7f1e0b5cf700 1 -- 172.21.15.188:0/114368891 wait complete. 2023-11-15T04:13:55.694 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:56.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:55.690225+0000 mon.smithi038 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.188:0/114368891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:57.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:57.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:13:57.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:56.092712+0000 mgr.smithi038.bnbogz (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:58.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 -- 172.21.15.188:0/3004931246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 msgr2=0x7f0df40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:58.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 --2- 172.21.15.188:0/3004931246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df40f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f0de8009fb0 tx=0x7f0de8009b30).stop 2023-11-15T04:13:58.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 -- 172.21.15.188:0/3004931246 shutdown_connections 2023-11-15T04:13:58.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 --2- 172.21.15.188:0/3004931246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df40f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:58.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 -- 172.21.15.188:0/3004931246 >> 172.21.15.188:0/3004931246 conn(0x7f0df40f0b70 msgr2=0x7f0df40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:58.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 -- 172.21.15.188:0/3004931246 shutdown_connections 2023-11-15T04:13:58.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.303+0000 7f0df92c8700 1 -- 172.21.15.188:0/3004931246 wait complete. 2023-11-15T04:13:58.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df92c8700 1 Processor -- start 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df92c8700 1 -- start start 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df92c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df92c8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4106c20 con 0x7f0df40f5410 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47792/0 (socket says 172.21.15.188:47792) 2023-11-15T04:13:58.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.304+0000 7f0df2ffd700 1 -- 172.21.15.188:0/468296387 learned_addr learned my addr 172.21.15.188:0/468296387 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:13:58.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0df2ffd700 1 -- 172.21.15.188:0/468296387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8005040 con 0x7f0df40f5410 2023-11-15T04:13:58.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0df2ffd700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0de8014000 tx=0x7f0de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:58.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de80095f0 con 0x7f0df40f5410 2023-11-15T04:13:58.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df4106e20 con 0x7f0df40f5410 2023-11-15T04:13:58.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0de8007e70 con 0x7f0df40f5410 2023-11-15T04:13:58.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de8014ea0 con 0x7f0df40f5410 2023-11-15T04:13:58.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.305+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df41072b0 con 0x7f0df40f5410 2023-11-15T04:13:58.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.306+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0de8025070 con 0x7f0df40f5410 2023-11-15T04:13:58.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.306+0000 7f0ddb7fe700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 0x7f0ddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:13:58.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.306+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0de8040980 con 0x7f0df40f5410 2023-11-15T04:13:58.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.306+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df4045ad0 con 0x7f0df40f5410 2023-11-15T04:13:58.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.306+0000 7f0df27fc700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 0x7f0ddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:13:58.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.307+0000 7f0df27fc700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 0x7f0ddc032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0de40096a0 tx=0x7f0de4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:13:58.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.311+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0de8026350 con 0x7f0df40f5410 2023-11-15T04:13:58.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.515+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0df40faf50 con 0x7f0df40f5410 2023-11-15T04:13:58.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.516+0000 7f0ddb7fe700 1 -- 172.21.15.188:0/468296387 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0de8012330 con 0x7f0df40f5410 2023-11-15T04:13:58.517 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:13:58.517 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:13:58.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 msgr2=0x7f0ddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:58.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 0x7f0ddc032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0de40096a0 tx=0x7f0de4006b40).stop 2023-11-15T04:13:58.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 msgr2=0x7f0df4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:13:58.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0de8014000 tx=0x7f0de800b040).stop 2023-11-15T04:13:58.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 shutdown_connections 2023-11-15T04:13:58.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0ddc02ff60 0x7f0ddc032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:58.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 --2- 172.21.15.188:0/468296387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df40f5410 0x7f0df4106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:13:58.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 >> 172.21.15.188:0/468296387 conn(0x7f0df40f0b70 msgr2=0x7f0df40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:13:58.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 shutdown_connections 2023-11-15T04:13:58.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:13:58.518+0000 7f0df92c8700 1 -- 172.21.15.188:0/468296387 wait complete. 2023-11-15T04:13:58.521 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:13:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:13:58.093031+0000 mgr.smithi038.bnbogz (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:13:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:13:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:13:58.517556+0000 mon.smithi038 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.188:0/468296387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:13:59.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:13:59.997 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:01.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 -- 172.21.15.188:0/1737481963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 msgr2=0x7fd0100f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:01.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 --2- 172.21.15.188:0/1737481963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd0100f4f20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fcff8009fb0 tx=0x7fcff8009b30).stop 2023-11-15T04:14:01.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 -- 172.21.15.188:0/1737481963 shutdown_connections 2023-11-15T04:14:01.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 --2- 172.21.15.188:0/1737481963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd0100f4f20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:01.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 -- 172.21.15.188:0/1737481963 >> 172.21.15.188:0/1737481963 conn(0x7fd0100f0260 msgr2=0x7fd0100f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 -- 172.21.15.188:0/1737481963 shutdown_connections 2023-11-15T04:14:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.193+0000 7fd01504a700 1 -- 172.21.15.188:0/1737481963 wait complete. 2023-11-15T04:14:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd01504a700 1 Processor -- start 2023-11-15T04:14:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd01504a700 1 -- start start 2023-11-15T04:14:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd01504a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:01.197 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd01504a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd010106c10 con 0x7fd0100f4b00 2023-11-15T04:14:01.198 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd00ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:01.198 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd00ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47808/0 (socket says 172.21.15.188:47808) 2023-11-15T04:14:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd00ed9d700 1 -- 172.21.15.188:0/1990807265 learned_addr learned my addr 172.21.15.188:0/1990807265 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.194+0000 7fd00ed9d700 1 -- 172.21.15.188:0/1990807265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8005040 con 0x7fd0100f4b00 2023-11-15T04:14:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd00ed9d700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fcff8013510 tx=0x7fcff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff8009320 con 0x7fd0100f4b00 2023-11-15T04:14:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcff8013f00 con 0x7fd0100f4b00 2023-11-15T04:14:01.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd010106e10 con 0x7fd0100f4b00 2023-11-15T04:14:01.200 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff8014d90 con 0x7fd0100f4b00 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.195+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0101072a0 con 0x7fd0100f4b00 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcff80095f0 con 0x7fd0100f4b00 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd010045ad0 con 0x7fd0100f4b00 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd0077fe700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 0x7fcffc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcff8040fc0 con 0x7fd0100f4b00 2023-11-15T04:14:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd00e59c700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 0x7fcffc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:01.202 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.196+0000 7fd00e59c700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 0x7fcffc032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd0000096a0 tx=0x7fd000006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:01.202 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.200+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcff8014420 con 0x7fd0100f4b00 2023-11-15T04:14:01.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.404+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0101080f0 con 0x7fd0100f4b00 2023-11-15T04:14:01.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.405+0000 7fd0077fe700 1 -- 172.21.15.188:0/1990807265 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcff8010230 con 0x7fd0100f4b00 2023-11-15T04:14:01.406 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:01.406 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:01.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 msgr2=0x7fcffc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 0x7fcffc032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd0000096a0 tx=0x7fd000006b40).stop 2023-11-15T04:14:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 msgr2=0x7fd010106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fcff8013510 tx=0x7fcff800b040).stop 2023-11-15T04:14:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 shutdown_connections 2023-11-15T04:14:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fcffc02ff60 0x7fcffc032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 --2- 172.21.15.188:0/1990807265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0100f4b00 0x7fd010106720 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 >> 172.21.15.188:0/1990807265 conn(0x7fd0100f0260 msgr2=0x7fd0100f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 shutdown_connections 2023-11-15T04:14:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:01.407+0000 7fd01504a700 1 -- 172.21.15.188:0/1990807265 wait complete. 2023-11-15T04:14:01.410 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:00.093399+0000 mgr.smithi038.bnbogz (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:02.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:01.406572+0000 mon.smithi038 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.188:0/1990807265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:02.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:02.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:03.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:02.093717+0000 mgr.smithi038.bnbogz (mgr.14164) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 -- 172.21.15.188:0/1415362337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 msgr2=0x7f6e180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/1415362337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e180f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6e00009fb0 tx=0x7f6e00009b30).stop 2023-11-15T04:14:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 -- 172.21.15.188:0/1415362337 shutdown_connections 2023-11-15T04:14:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/1415362337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e180f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 -- 172.21.15.188:0/1415362337 >> 172.21.15.188:0/1415362337 conn(0x7f6e180f0b70 msgr2=0x7f6e180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 -- 172.21.15.188:0/1415362337 shutdown_connections 2023-11-15T04:14:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.019+0000 7f6e1caa0700 1 -- 172.21.15.188:0/1415362337 wait complete. 2023-11-15T04:14:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.020+0000 7f6e1caa0700 1 Processor -- start 2023-11-15T04:14:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.020+0000 7f6e1caa0700 1 -- start start 2023-11-15T04:14:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.020+0000 7f6e1caa0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.020+0000 7f6e1caa0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18060220 con 0x7f6e180f5410 2023-11-15T04:14:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.020+0000 7f6e1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47820/0 (socket says 172.21.15.188:47820) 2023-11-15T04:14:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1659c700 1 -- 172.21.15.188:0/3970119627 learned_addr learned my addr 172.21.15.188:0/3970119627 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1659c700 1 -- 172.21.15.188:0/3970119627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00005040 con 0x7f6e180f5410 2023-11-15T04:14:04.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1659c700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6e00014000 tx=0x7f6e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:04.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e000095f0 con 0x7f6e180f5410 2023-11-15T04:14:04.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e00007e70 con 0x7f6e180f5410 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e18060420 con 0x7f6e180f5410 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e00014ea0 con 0x7f6e180f5410 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.021+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e18109ad0 con 0x7f6e180f5410 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.022+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e00025070 con 0x7f6e180f5410 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.022+0000 7f6e0effd700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 0x7f6e04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.022+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e000409b0 con 0x7f6e180f5410 2023-11-15T04:14:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.022+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e18059100 con 0x7f6e180f5410 2023-11-15T04:14:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.022+0000 7f6e15d9b700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 0x7f6e04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.023+0000 7f6e15d9b700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 0x7f6e04032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6e080096a0 tx=0x7f6e08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:04.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.027+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e00026350 con 0x7f6e180f5410 2023-11-15T04:14:04.234 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.234+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e1810a1a0 con 0x7f6e180f5410 2023-11-15T04:14:04.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.234+0000 7f6e0effd700 1 -- 172.21.15.188:0/3970119627 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e00012330 con 0x7f6e180f5410 2023-11-15T04:14:04.236 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:04.236 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 msgr2=0x7f6e04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 0x7f6e04032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6e080096a0 tx=0x7f6e08006b40).stop 2023-11-15T04:14:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 msgr2=0x7f6e18109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6e00014000 tx=0x7f6e0000b040).stop 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 shutdown_connections 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6e0402ff60 0x7f6e04032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 --2- 172.21.15.188:0/3970119627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e180f5410 0x7f6e18109420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 >> 172.21.15.188:0/3970119627 conn(0x7f6e180f0b70 msgr2=0x7f6e180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 shutdown_connections 2023-11-15T04:14:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:04.236+0000 7f6e1caa0700 1 -- 172.21.15.188:0/3970119627 wait complete. 2023-11-15T04:14:04.239 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:04.094085+0000 mgr.smithi038.bnbogz (mgr.14164) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:04.235909+0000 mon.smithi038 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.188:0/3970119627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:05.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:05.652 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:06.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.809+0000 7f3a57548700 1 -- 172.21.15.188:0/1998629687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 msgr2=0x7f3a500f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:06.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.809+0000 7f3a57548700 1 --2- 172.21.15.188:0/1998629687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500f5810 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f3a40009fb0 tx=0x7f3a40009b30).stop 2023-11-15T04:14:06.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.809+0000 7f3a57548700 1 -- 172.21.15.188:0/1998629687 shutdown_connections 2023-11-15T04:14:06.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.809+0000 7f3a57548700 1 --2- 172.21.15.188:0/1998629687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500f5810 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:06.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.809+0000 7f3a57548700 1 -- 172.21.15.188:0/1998629687 >> 172.21.15.188:0/1998629687 conn(0x7f3a500f0b70 msgr2=0x7f3a500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:06.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 -- 172.21.15.188:0/1998629687 shutdown_connections 2023-11-15T04:14:06.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 -- 172.21.15.188:0/1998629687 wait complete. 2023-11-15T04:14:06.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 Processor -- start 2023-11-15T04:14:06.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 -- start start 2023-11-15T04:14:06.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:06.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a57548700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a500fb800 con 0x7f3a500f53f0 2023-11-15T04:14:06.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.810+0000 7f3a552e4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:06.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a552e4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47842/0 (socket says 172.21.15.188:47842) 2023-11-15T04:14:06.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a552e4700 1 -- 172.21.15.188:0/703037719 learned_addr learned my addr 172.21.15.188:0/703037719 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:06.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a552e4700 1 -- 172.21.15.188:0/703037719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40005040 con 0x7f3a500f53f0 2023-11-15T04:14:06.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a552e4700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3a40013510 tx=0x7f3a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:06.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a40009320 con 0x7f3a500f53f0 2023-11-15T04:14:06.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a40013f00 con 0x7f3a500f53f0 2023-11-15T04:14:06.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a500fb9a0 con 0x7f3a500f53f0 2023-11-15T04:14:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a40014d90 con 0x7f3a500f53f0 2023-11-15T04:14:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.811+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a500fbe30 con 0x7f3a500f53f0 2023-11-15T04:14:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.812+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a400263a0 con 0x7f3a500f53f0 2023-11-15T04:14:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.812+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a50059100 con 0x7f3a500f53f0 2023-11-15T04:14:06.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.812+0000 7f3a45ffb700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:06.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.812+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a40041380 con 0x7f3a500f53f0 2023-11-15T04:14:06.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.813+0000 7f3a54ae3700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:06.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.813+0000 7f3a54ae3700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 0x7f3a3c032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3a4c0096a0 tx=0x7f3a4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:06.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:06.817+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a40014420 con 0x7f3a500f53f0 2023-11-15T04:14:07.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.024+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a500fae40 con 0x7f3a500f53f0 2023-11-15T04:14:07.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.025+0000 7f3a45ffb700 1 -- 172.21.15.188:0/703037719 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a4001cce0 con 0x7f3a500f53f0 2023-11-15T04:14:07.025 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:07.026 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:07.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 msgr2=0x7f3a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:07.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 0x7f3a3c032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3a4c0096a0 tx=0x7f3a4c006b40).stop 2023-11-15T04:14:07.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 msgr2=0x7f3a500fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:07.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3a40013510 tx=0x7f3a4000b040).stop 2023-11-15T04:14:07.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 shutdown_connections 2023-11-15T04:14:07.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3a3c02ff60 0x7f3a3c032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:07.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 --2- 172.21.15.188:0/703037719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a500f53f0 0x7f3a500fd380 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:07.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 >> 172.21.15.188:0/703037719 conn(0x7f3a500f0b70 msgr2=0x7f3a5005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:07.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 shutdown_connections 2023-11-15T04:14:07.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:07.026+0000 7f3a57548700 1 -- 172.21.15.188:0/703037719 wait complete. 2023-11-15T04:14:07.029 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:06.094335+0000 mgr.smithi038.bnbogz (mgr.14164) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:07.026003+0000 mon.smithi038 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.188:0/703037719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:08.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:08.438 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:08.094598+0000 mgr.smithi038.bnbogz (mgr.14164) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:09.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 -- 172.21.15.188:0/628916994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 msgr2=0x7fab940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:09.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 --2- 172.21.15.188:0/628916994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab940f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fab8c009960 tx=0x7fab8c004d10).stop 2023-11-15T04:14:09.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 -- 172.21.15.188:0/628916994 shutdown_connections 2023-11-15T04:14:09.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 --2- 172.21.15.188:0/628916994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab940f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:09.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 -- 172.21.15.188:0/628916994 >> 172.21.15.188:0/628916994 conn(0x7fab940f0b70 msgr2=0x7fab940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:09.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 -- 172.21.15.188:0/628916994 shutdown_connections 2023-11-15T04:14:09.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.630+0000 7fab9c32d700 1 -- 172.21.15.188:0/628916994 wait complete. 2023-11-15T04:14:09.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9c32d700 1 Processor -- start 2023-11-15T04:14:09.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9c32d700 1 -- start start 2023-11-15T04:14:09.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9c32d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:09.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9c32d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94060240 con 0x7fab940f5410 2023-11-15T04:14:09.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9a0c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:09.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9a0c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37750/0 (socket says 172.21.15.188:37750) 2023-11-15T04:14:09.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.631+0000 7fab9a0c9700 1 -- 172.21.15.188:0/3846378627 learned_addr learned my addr 172.21.15.188:0/3846378627 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:09.636 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab9a0c9700 1 -- 172.21.15.188:0/3846378627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c009640 con 0x7fab940f5410 2023-11-15T04:14:09.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab9a0c9700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fab8c004b70 tx=0x7fab8c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:09.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab8c018070 con 0x7fab940f5410 2023-11-15T04:14:09.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab94060440 con 0x7fab940f5410 2023-11-15T04:14:09.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab8c00b870 con 0x7fab940f5410 2023-11-15T04:14:09.637 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab8c01c560 con 0x7fab940f5410 2023-11-15T04:14:09.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.632+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab94109ae0 con 0x7fab940f5410 2023-11-15T04:14:09.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.633+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94045ad0 con 0x7fab940f5410 2023-11-15T04:14:09.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.633+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab8c013630 con 0x7fab940f5410 2023-11-15T04:14:09.638 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.633+0000 7fab8affd700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 0x7fab80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:09.639 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.633+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab8c0411d0 con 0x7fab940f5410 2023-11-15T04:14:09.639 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.633+0000 7fab998c8700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 0x7fab80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:09.639 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.634+0000 7fab998c8700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 0x7fab80032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:09.640 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.638+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab8c004810 con 0x7fab940f5410 2023-11-15T04:14:09.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.842+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab940fae60 con 0x7fab940f5410 2023-11-15T04:14:09.843 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.842+0000 7fab8affd700 1 -- 172.21.15.188:0/3846378627 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab8c012380 con 0x7fab940f5410 2023-11-15T04:14:09.843 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:09.844 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:09.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 msgr2=0x7fab80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:09.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 0x7fab80032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).stop 2023-11-15T04:14:09.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 msgr2=0x7fab94109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:09.845 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fab8c004b70 tx=0x7fab8c004d10).stop 2023-11-15T04:14:09.846 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 shutdown_connections 2023-11-15T04:14:09.846 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fab8002ff60 0x7fab80032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:09.846 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 --2- 172.21.15.188:0/3846378627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab940f5410 0x7fab94109430 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:09.846 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 >> 172.21.15.188:0/3846378627 conn(0x7fab940f0b70 msgr2=0x7fab940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:09.847 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 shutdown_connections 2023-11-15T04:14:09.847 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:09.844+0000 7fab9c32d700 1 -- 172.21.15.188:0/3846378627 wait complete. 2023-11-15T04:14:09.847 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:10.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:10 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:09.843780+0000 mon.smithi038 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.188:0/3846378627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:11.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:11.261 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:10.094833+0000 mgr.smithi038.bnbogz (mgr.14164) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:12.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 -- 172.21.15.188:0/1835463824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 msgr2=0x7fc35c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:12.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1835463824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc350009fb0 tx=0x7fc350009b30).stop 2023-11-15T04:14:12.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 -- 172.21.15.188:0/1835463824 shutdown_connections 2023-11-15T04:14:12.480 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1835463824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:12.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 -- 172.21.15.188:0/1835463824 >> 172.21.15.188:0/1835463824 conn(0x7fc35c0f1c70 msgr2=0x7fc35c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:12.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 -- 172.21.15.188:0/1835463824 shutdown_connections 2023-11-15T04:14:12.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.477+0000 7fc360ea1700 1 -- 172.21.15.188:0/1835463824 wait complete. 2023-11-15T04:14:12.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc360ea1700 1 Processor -- start 2023-11-15T04:14:12.481 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc360ea1700 1 -- start start 2023-11-15T04:14:12.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc360ea1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:12.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc360ea1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c106c20 con 0x7fc35c0f6510 2023-11-15T04:14:12.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc35a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:12.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc35a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37772/0 (socket says 172.21.15.188:37772) 2023-11-15T04:14:12.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.478+0000 7fc35a59c700 1 -- 172.21.15.188:0/1668096434 learned_addr learned my addr 172.21.15.188:0/1668096434 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:12.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc35a59c700 1 -- 172.21.15.188:0/1668096434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350005040 con 0x7fc35c0f6510 2023-11-15T04:14:12.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc35a59c700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc350013510 tx=0x7fc35000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:12.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc350009320 con 0x7fc35c0f6510 2023-11-15T04:14:12.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c106e20 con 0x7fc35c0f6510 2023-11-15T04:14:12.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc350013f00 con 0x7fc35c0f6510 2023-11-15T04:14:12.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc350014d90 con 0x7fc35c0f6510 2023-11-15T04:14:12.485 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.479+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c1072b0 con 0x7fc35c0f6510 2023-11-15T04:14:12.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.480+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3500095f0 con 0x7fc35c0f6510 2023-11-15T04:14:12.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.480+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c045ad0 con 0x7fc35c0f6510 2023-11-15T04:14:12.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.480+0000 7fc34affd700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 0x7fc344032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:12.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.480+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc350040fd0 con 0x7fc35c0f6510 2023-11-15T04:14:12.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.480+0000 7fc359d9b700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 0x7fc344032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:12.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.481+0000 7fc359d9b700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 0x7fc344032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc34c0096a0 tx=0x7fc34c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:12.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.485+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc350014420 con 0x7fc35c0f6510 2023-11-15T04:14:12.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.688+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc35c0fbf60 con 0x7fc35c0f6510 2023-11-15T04:14:12.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.689+0000 7fc34affd700 1 -- 172.21.15.188:0/1668096434 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc35001d7f0 con 0x7fc35c0f6510 2023-11-15T04:14:12.690 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:12.690 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:12.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 msgr2=0x7fc344032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:12.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 0x7fc344032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc34c0096a0 tx=0x7fc34c006b40).stop 2023-11-15T04:14:12.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 msgr2=0x7fc35c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:12.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc350013510 tx=0x7fc35000b040).stop 2023-11-15T04:14:12.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 shutdown_connections 2023-11-15T04:14:12.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc34402ff60 0x7fc344032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:12.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 --2- 172.21.15.188:0/1668096434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc35c0f6510 0x7fc35c106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:12.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 >> 172.21.15.188:0/1668096434 conn(0x7fc35c0f1c70 msgr2=0x7fc35c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:12.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 shutdown_connections 2023-11-15T04:14:12.693 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:12.690+0000 7fc360ea1700 1 -- 172.21.15.188:0/1668096434 wait complete. 2023-11-15T04:14:12.693 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:12.095156+0000 mgr.smithi038.bnbogz (mgr.14164) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:13.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:12.690162+0000 mon.smithi038 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.188:0/1668096434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:14.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:14.081 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:14.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:14.095433+0000 mgr.smithi038.bnbogz (mgr.14164) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:15.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 -- 172.21.15.188:0/3207815342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 msgr2=0x7f84180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:15.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 --2- 172.21.15.188:0/3207815342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84180f66c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f8408009fb0 tx=0x7f8408009b30).stop 2023-11-15T04:14:15.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 -- 172.21.15.188:0/3207815342 shutdown_connections 2023-11-15T04:14:15.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 --2- 172.21.15.188:0/3207815342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84180f66c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:15.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 -- 172.21.15.188:0/3207815342 >> 172.21.15.188:0/3207815342 conn(0x7f84180f1aa0 msgr2=0x7f84180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:15.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 -- 172.21.15.188:0/3207815342 shutdown_connections 2023-11-15T04:14:15.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.308+0000 7f8417fff700 1 -- 172.21.15.188:0/3207815342 wait complete. 2023-11-15T04:14:15.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8417fff700 1 Processor -- start 2023-11-15T04:14:15.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8417fff700 1 -- start start 2023-11-15T04:14:15.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8417fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:15.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8417fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84181069a0 con 0x7f84180f62a0 2023-11-15T04:14:15.313 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8416ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:15.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8416ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37784/0 (socket says 172.21.15.188:37784) 2023-11-15T04:14:15.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.309+0000 7f8416ffd700 1 -- 172.21.15.188:0/3345866346 learned_addr learned my addr 172.21.15.188:0/3345866346 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:15.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f8416ffd700 1 -- 172.21.15.188:0/3345866346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8408005040 con 0x7f84180f62a0 2023-11-15T04:14:15.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f8416ffd700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8408013510 tx=0x7f840800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:15.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408009320 con 0x7f84180f62a0 2023-11-15T04:14:15.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8418106ba0 con 0x7f84180f62a0 2023-11-15T04:14:15.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8408013f00 con 0x7f84180f62a0 2023-11-15T04:14:15.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408014dc0 con 0x7f84180f62a0 2023-11-15T04:14:15.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.310+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8418107030 con 0x7f84180f62a0 2023-11-15T04:14:15.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.311+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84080095f0 con 0x7f84180f62a0 2023-11-15T04:14:15.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.311+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8418045ad0 con 0x7f84180f62a0 2023-11-15T04:14:15.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.311+0000 7f83ff7fe700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 0x7f8400032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:15.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.311+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8408041000 con 0x7f84180f62a0 2023-11-15T04:14:15.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.311+0000 7f84167fc700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 0x7f8400032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:15.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.312+0000 7f84167fc700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 0x7f8400032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f840c0096a0 tx=0x7f840c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:15.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.316+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8408014420 con 0x7f84180f62a0 2023-11-15T04:14:15.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.520+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8418107ee0 con 0x7f84180f62a0 2023-11-15T04:14:15.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.521+0000 7f83ff7fe700 1 -- 172.21.15.188:0/3345866346 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8408010230 con 0x7f84180f62a0 2023-11-15T04:14:15.522 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:15.522 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:15.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.522+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 msgr2=0x7f8400032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:15.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.522+0000 7f8417fff700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 0x7f8400032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f840c0096a0 tx=0x7f840c006b40).stop 2023-11-15T04:14:15.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.522+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 msgr2=0x7f84181064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:15.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8408013510 tx=0x7f840800b040).stop 2023-11-15T04:14:15.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 shutdown_connections 2023-11-15T04:14:15.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f840002ff60 0x7f8400032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:15.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 --2- 172.21.15.188:0/3345866346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84180f62a0 0x7f84181064b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:15.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 >> 172.21.15.188:0/3345866346 conn(0x7f84180f1aa0 msgr2=0x7f84180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:15.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 shutdown_connections 2023-11-15T04:14:15.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:15.523+0000 7f8417fff700 1 -- 172.21.15.188:0/3345866346 wait complete. 2023-11-15T04:14:15.525 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:15.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:15 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:15.522203+0000 mon.smithi038 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.188:0/3345866346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:16.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:16 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:16.095755+0000 mgr.smithi038.bnbogz (mgr.14164) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:16.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:16.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:18.202 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 -- 172.21.15.188:0/124757389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 msgr2=0x7f86980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:18.202 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 --2- 172.21.15.188:0/124757389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86980f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8688009fb0 tx=0x7f8688009b30).stop 2023-11-15T04:14:18.202 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 -- 172.21.15.188:0/124757389 shutdown_connections 2023-11-15T04:14:18.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 --2- 172.21.15.188:0/124757389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86980f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:18.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 -- 172.21.15.188:0/124757389 >> 172.21.15.188:0/124757389 conn(0x7f86980f0b70 msgr2=0x7f86980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:18.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 -- 172.21.15.188:0/124757389 shutdown_connections 2023-11-15T04:14:18.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 -- 172.21.15.188:0/124757389 wait complete. 2023-11-15T04:14:18.203 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.200+0000 7f86a07f7700 1 Processor -- start 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f86a07f7700 1 -- start start 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f86a07f7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f86a07f7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8698060280 con 0x7f86980f5410 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f869e593700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f869e593700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:37816/0 (socket says 172.21.15.188:37816) 2023-11-15T04:14:18.204 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f869e593700 1 -- 172.21.15.188:0/1289232042 learned_addr learned my addr 172.21.15.188:0/1289232042 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:18.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f869e593700 1 -- 172.21.15.188:0/1289232042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688005040 con 0x7f86980f5410 2023-11-15T04:14:18.205 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.201+0000 7f869e593700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8688013510 tx=0x7f868800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:18.207 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8688009320 con 0x7f86980f5410 2023-11-15T04:14:18.207 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8688013f00 con 0x7f86980f5410 2023-11-15T04:14:18.207 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8688014d90 con 0x7f86980f5410 2023-11-15T04:14:18.207 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8698060480 con 0x7f86980f5410 2023-11-15T04:14:18.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8698109b20 con 0x7f86980f5410 2023-11-15T04:14:18.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86880095f0 con 0x7f86980f5410 2023-11-15T04:14:18.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.202+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8698045ad0 con 0x7f86980f5410 2023-11-15T04:14:18.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.203+0000 7f868effd700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 0x7f8684032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:18.208 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.203+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8688040f90 con 0x7f86980f5410 2023-11-15T04:14:18.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.203+0000 7f869dd92700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 0x7f8684032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:18.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.203+0000 7f869dd92700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 0x7f8684032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f86940096a0 tx=0x7f8694006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:18.209 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.208+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8688014420 con 0x7f86980f5410 2023-11-15T04:14:18.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.414+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f869810a340 con 0x7f86980f5410 2023-11-15T04:14:18.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.415+0000 7f868effd700 1 -- 172.21.15.188:0/1289232042 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f868801d7f0 con 0x7f86980f5410 2023-11-15T04:14:18.417 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:18.417 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:18.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 msgr2=0x7f8684032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:18.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 0x7f8684032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f86940096a0 tx=0x7f8694006b40).stop 2023-11-15T04:14:18.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 msgr2=0x7f86981093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:18.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8688013510 tx=0x7f868800b040).stop 2023-11-15T04:14:18.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 shutdown_connections 2023-11-15T04:14:18.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f868402ffb0 0x7f8684032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:18.421 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 --2- 172.21.15.188:0/1289232042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86980f5410 0x7f86981093f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:18.421 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 >> 172.21.15.188:0/1289232042 conn(0x7f86980f0b70 msgr2=0x7f86980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:18.422 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 shutdown_connections 2023-11-15T04:14:18.422 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:18.416+0000 7f86a07f7700 1 -- 172.21.15.188:0/1289232042 wait complete. 2023-11-15T04:14:18.423 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:19.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:18.096103+0000 mgr.smithi038.bnbogz (mgr.14164) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:19.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:18.415953+0000 mon.smithi038 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.188:0/1289232042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:19.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:19.808 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:21.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 -- 172.21.15.188:0/715067867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 msgr2=0x7fa2a40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:21.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 --2- 172.21.15.188:0/715067867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a40f90f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fa298009fb0 tx=0x7fa298009b30).stop 2023-11-15T04:14:21.124 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 -- 172.21.15.188:0/715067867 shutdown_connections 2023-11-15T04:14:21.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 --2- 172.21.15.188:0/715067867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a40f90f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:21.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 -- 172.21.15.188:0/715067867 >> 172.21.15.188:0/715067867 conn(0x7fa2a40f0b70 msgr2=0x7fa2a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:21.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 -- 172.21.15.188:0/715067867 shutdown_connections 2023-11-15T04:14:21.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.122+0000 7fa2ab294700 1 -- 172.21.15.188:0/715067867 wait complete. 2023-11-15T04:14:21.125 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2ab294700 1 Processor -- start 2023-11-15T04:14:21.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2ab294700 1 -- start start 2023-11-15T04:14:21.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2ab294700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:21.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2ab294700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a4106bd0 con 0x7fa2a4060280 2023-11-15T04:14:21.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2a9030700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:21.126 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2a9030700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50310/0 (socket says 172.21.15.188:50310) 2023-11-15T04:14:21.127 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.123+0000 7fa2a9030700 1 -- 172.21.15.188:0/553671877 learned_addr learned my addr 172.21.15.188:0/553671877 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:21.128 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa2a9030700 1 -- 172.21.15.188:0/553671877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa298005040 con 0x7fa2a4060280 2023-11-15T04:14:21.128 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa2a9030700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fa298013510 tx=0x7fa29800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:21.129 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa298009320 con 0x7fa2a4060280 2023-11-15T04:14:21.129 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a4106dd0 con 0x7fa2a4060280 2023-11-15T04:14:21.129 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa298013f00 con 0x7fa2a4060280 2023-11-15T04:14:21.129 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa298014dc0 con 0x7fa2a4060280 2023-11-15T04:14:21.129 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.124+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a4107260 con 0x7fa2a4060280 2023-11-15T04:14:21.130 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.125+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa298014420 con 0x7fa2a4060280 2023-11-15T04:14:21.130 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.125+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a4045ad0 con 0x7fa2a4060280 2023-11-15T04:14:21.130 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.125+0000 7fa295ffb700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 0x7fa290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:21.130 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.125+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2980410e0 con 0x7fa2a4060280 2023-11-15T04:14:21.131 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.125+0000 7fa2a882f700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 0x7fa290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:21.131 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.126+0000 7fa2a882f700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 0x7fa290032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa2a00096a0 tx=0x7fa2a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:21.131 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.130+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2980095f0 con 0x7fa2a4060280 2023-11-15T04:14:21.334 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.333+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2a41080b0 con 0x7fa2a4060280 2023-11-15T04:14:21.335 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.334+0000 7fa295ffb700 1 -- 172.21.15.188:0/553671877 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa29801cb10 con 0x7fa2a4060280 2023-11-15T04:14:21.335 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:21.335 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:21.336 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 msgr2=0x7fa290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:21.336 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 0x7fa290032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa2a00096a0 tx=0x7fa2a0006b40).stop 2023-11-15T04:14:21.337 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 msgr2=0x7fa2a41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:21.337 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fa298013510 tx=0x7fa29800b040).stop 2023-11-15T04:14:21.337 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 shutdown_connections 2023-11-15T04:14:21.337 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa29002ff60 0x7fa290032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:21.338 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 --2- 172.21.15.188:0/553671877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a4060280 0x7fa2a41066e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:21.338 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 >> 172.21.15.188:0/553671877 conn(0x7fa2a40f0b70 msgr2=0x7fa2a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:21.338 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 shutdown_connections 2023-11-15T04:14:21.338 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:21.336+0000 7fa2ab294700 1 -- 172.21.15.188:0/553671877 wait complete. 2023-11-15T04:14:21.339 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:20.096475+0000 mgr.smithi038.bnbogz (mgr.14164) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:22.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:21.335511+0000 mon.smithi038 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.188:0/553671877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:22.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:22.761 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:23.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:22.096837+0000 mgr.smithi038.bnbogz (mgr.14164) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:23.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- 172.21.15.188:0/4223834443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 msgr2=0x7f94ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:23.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 --2- 172.21.15.188:0/4223834443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac0f90f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f94a0009fb0 tx=0x7f94a0009b30).stop 2023-11-15T04:14:23.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- 172.21.15.188:0/4223834443 shutdown_connections 2023-11-15T04:14:23.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 --2- 172.21.15.188:0/4223834443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac0f90f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:23.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- 172.21.15.188:0/4223834443 >> 172.21.15.188:0/4223834443 conn(0x7f94ac0f0b70 msgr2=0x7f94ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:23.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- 172.21.15.188:0/4223834443 shutdown_connections 2023-11-15T04:14:23.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- 172.21.15.188:0/4223834443 wait complete. 2023-11-15T04:14:23.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 Processor -- start 2023-11-15T04:14:23.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.979+0000 7f94b268b700 1 -- start start 2023-11-15T04:14:23.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94b268b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:23.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94b268b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac106bd0 con 0x7f94ac060280 2023-11-15T04:14:23.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94abfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:23.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94abfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50328/0 (socket says 172.21.15.188:50328) 2023-11-15T04:14:23.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94abfff700 1 -- 172.21.15.188:0/4058390118 learned_addr learned my addr 172.21.15.188:0/4058390118 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:23.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94abfff700 1 -- 172.21.15.188:0/4058390118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0005040 con 0x7f94ac060280 2023-11-15T04:14:23.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94abfff700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f94a0013510 tx=0x7f94a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:23.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.980+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a0009320 con 0x7f94ac060280 2023-11-15T04:14:23.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94a0013f00 con 0x7f94ac060280 2023-11-15T04:14:23.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac106dd0 con 0x7f94ac060280 2023-11-15T04:14:23.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a0014dc0 con 0x7f94ac060280 2023-11-15T04:14:23.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac107260 con 0x7f94ac060280 2023-11-15T04:14:23.986 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94a00095f0 con 0x7f94ac060280 2023-11-15T04:14:23.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac045ad0 con 0x7f94ac060280 2023-11-15T04:14:23.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94a8ff9700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 0x7f9494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:23.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.981+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94a0041000 con 0x7f94ac060280 2023-11-15T04:14:23.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.982+0000 7f94ab7fe700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 0x7f9494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:23.987 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.982+0000 7f94ab7fe700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 0x7f9494032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f949c0096a0 tx=0x7f949c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:23.988 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:23.986+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94a001c790 con 0x7f94ac060280 2023-11-15T04:14:24.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.190+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94ac108110 con 0x7f94ac060280 2023-11-15T04:14:24.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.191+0000 7f94a8ff9700 1 -- 172.21.15.188:0/4058390118 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94a0010230 con 0x7f94ac060280 2023-11-15T04:14:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:24.192 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 msgr2=0x7f9494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 0x7f9494032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f949c0096a0 tx=0x7f949c006b40).stop 2023-11-15T04:14:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 msgr2=0x7f94ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f94a0013510 tx=0x7f94a000b040).stop 2023-11-15T04:14:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 shutdown_connections 2023-11-15T04:14:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f949402ff60 0x7f9494032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 --2- 172.21.15.188:0/4058390118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94ac060280 0x7f94ac1066e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:24.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 >> 172.21.15.188:0/4058390118 conn(0x7f94ac0f0b70 msgr2=0x7f94ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:24.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 shutdown_connections 2023-11-15T04:14:24.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:24.193+0000 7f94b268b700 1 -- 172.21.15.188:0/4058390118 wait complete. 2023-11-15T04:14:24.195 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:25.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:25.581 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:24.097101+0000 mgr.smithi038.bnbogz (mgr.14164) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:24.192400+0000 mon.smithi038 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.188:0/4058390118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:26.948 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 -- 172.21.15.188:0/3729595218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 msgr2=0x7fd5180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:26.949 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 --2- 172.21.15.188:0/3729595218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5180f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fd50c009fb0 tx=0x7fd50c009b30).stop 2023-11-15T04:14:26.949 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 -- 172.21.15.188:0/3729595218 shutdown_connections 2023-11-15T04:14:26.949 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 --2- 172.21.15.188:0/3729595218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5180f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:26.949 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 -- 172.21.15.188:0/3729595218 >> 172.21.15.188:0/3729595218 conn(0x7fd5180f09a0 msgr2=0x7fd5180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 -- 172.21.15.188:0/3729595218 shutdown_connections 2023-11-15T04:14:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 -- 172.21.15.188:0/3729595218 wait complete. 2023-11-15T04:14:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.948+0000 7fd51de38700 1 Processor -- start 2023-11-15T04:14:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51de38700 1 -- start start 2023-11-15T04:14:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51de38700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:26.951 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51de38700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5181069d0 con 0x7fd5180f51a0 2023-11-15T04:14:26.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51ce36700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:26.952 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51ce36700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:50336/0 (socket says 172.21.15.188:50336) 2023-11-15T04:14:26.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.949+0000 7fd51ce36700 1 -- 172.21.15.188:0/90809889 learned_addr learned my addr 172.21.15.188:0/90809889 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:26.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd51ce36700 1 -- 172.21.15.188:0/90809889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c005040 con 0x7fd5180f51a0 2023-11-15T04:14:26.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd51ce36700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fd50c009830 tx=0x7fd50c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:26.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c009320 con 0x7fd5180f51a0 2023-11-15T04:14:26.953 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd50c0136c0 con 0x7fd5180f51a0 2023-11-15T04:14:26.954 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd518106bd0 con 0x7fd5180f51a0 2023-11-15T04:14:26.954 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c01ee90 con 0x7fd5180f51a0 2023-11-15T04:14:26.954 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.950+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd518107060 con 0x7fd5180f51a0 2023-11-15T04:14:26.954 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.951+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd50c01e420 con 0x7fd5180f51a0 2023-11-15T04:14:26.954 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.951+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd518045ad0 con 0x7fd5180f51a0 2023-11-15T04:14:26.955 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.951+0000 7fd5157fa700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 0x7fd5080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:26.955 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.951+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd50c040ff0 con 0x7fd5180f51a0 2023-11-15T04:14:26.955 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.951+0000 7fd517fff700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 0x7fd5080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:26.955 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.952+0000 7fd517fff700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 0x7fd5080323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:26.956 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:26.956+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd50c06d2b0 con 0x7fd5180f51a0 2023-11-15T04:14:27.158 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.159+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd518107f10 con 0x7fd5180f51a0 2023-11-15T04:14:27.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.159+0000 7fd5157fa700 1 -- 172.21.15.188:0/90809889 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd50c02c2d0 con 0x7fd5180f51a0 2023-11-15T04:14:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:27.160 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:27.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 msgr2=0x7fd5080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:27.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 0x7fd5080323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).stop 2023-11-15T04:14:27.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 msgr2=0x7fd5181064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:27.161 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fd50c009830 tx=0x7fd50c00b040).stop 2023-11-15T04:14:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 shutdown_connections 2023-11-15T04:14:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd50802ff10 0x7fd5080323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 --2- 172.21.15.188:0/90809889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5180f51a0 0x7fd5181064e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 >> 172.21.15.188:0/90809889 conn(0x7fd5180f09a0 msgr2=0x7fd51805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:27.162 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 shutdown_connections 2023-11-15T04:14:27.163 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:27.161+0000 7fd51de38700 1 -- 172.21.15.188:0/90809889 wait complete. 2023-11-15T04:14:27.163 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:27.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:26.097388+0000 mgr.smithi038.bnbogz (mgr.14164) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:27.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:27.159668+0000 mon.smithi038 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.188:0/90809889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:28.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:28.585 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:28.097722+0000 mgr.smithi038.bnbogz (mgr.14164) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:29.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 -- 172.21.15.188:0/50629930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 msgr2=0x7f6dbc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:29.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 --2- 172.21.15.188:0/50629930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f90f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f6dac009fb0 tx=0x7f6dac009b30).stop 2023-11-15T04:14:29.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 -- 172.21.15.188:0/50629930 shutdown_connections 2023-11-15T04:14:29.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 --2- 172.21.15.188:0/50629930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f90f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:29.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 -- 172.21.15.188:0/50629930 >> 172.21.15.188:0/50629930 conn(0x7f6dbc0f0b70 msgr2=0x7f6dbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 -- 172.21.15.188:0/50629930 shutdown_connections 2023-11-15T04:14:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.811+0000 7f6dc3693700 1 -- 172.21.15.188:0/50629930 wait complete. 2023-11-15T04:14:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc3693700 1 Processor -- start 2023-11-15T04:14:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc3693700 1 -- start start 2023-11-15T04:14:29.814 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc3693700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:29.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc3693700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0f4ff0 con 0x7f6dbc060280 2023-11-15T04:14:29.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc142f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:29.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc142f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57542/0 (socket says 172.21.15.188:57542) 2023-11-15T04:14:29.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc142f700 1 -- 172.21.15.188:0/1749676599 learned_addr learned my addr 172.21.15.188:0/1749676599 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:29.815 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.812+0000 7f6dc142f700 1 -- 172.21.15.188:0/1749676599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac005040 con 0x7f6dbc060280 2023-11-15T04:14:29.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6dc142f700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6dac013510 tx=0x7f6dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:29.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac009320 con 0x7f6dbc060280 2023-11-15T04:14:29.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc0f5190 con 0x7f6dbc060280 2023-11-15T04:14:29.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dac013f00 con 0x7f6dbc060280 2023-11-15T04:14:29.816 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac014dc0 con 0x7f6dbc060280 2023-11-15T04:14:29.817 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.813+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc0f5620 con 0x7f6dbc060280 2023-11-15T04:14:29.818 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6dac01d3d0 con 0x7f6dbc060280 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc059100 con 0x7f6dbc060280 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6db1ffb700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 0x7f6da8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6dac041300 con 0x7f6dbc060280 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6dc0c2e700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 0x7f6da8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.814+0000 7f6dc0c2e700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 0x7f6da8032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6db80096a0 tx=0x7f6db8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:29.819 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:29.819+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dac01ccf0 con 0x7f6dbc060280 2023-11-15T04:14:30.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.024+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6dbc0fbf50 con 0x7f6dbc060280 2023-11-15T04:14:30.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.024+0000 7f6db1ffb700 1 -- 172.21.15.188:0/1749676599 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6dac01cb10 con 0x7f6dbc060280 2023-11-15T04:14:30.025 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:30.025 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:30.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 msgr2=0x7f6da8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:30.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 0x7f6da8032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6db80096a0 tx=0x7f6db8006b40).stop 2023-11-15T04:14:30.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 msgr2=0x7f6dbc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:30.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6dac013510 tx=0x7f6dac00b040).stop 2023-11-15T04:14:30.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 shutdown_connections 2023-11-15T04:14:30.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6da802ff60 0x7f6da8032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:30.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 --2- 172.21.15.188:0/1749676599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6dbc060280 0x7f6dbc0f6b70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:30.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 >> 172.21.15.188:0/1749676599 conn(0x7f6dbc0f0b70 msgr2=0x7f6dbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:30.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 shutdown_connections 2023-11-15T04:14:30.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:30.026+0000 7f6dc3693700 1 -- 172.21.15.188:0/1749676599 wait complete. 2023-11-15T04:14:30.028 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:30.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:29.686975+0000 mon.smithi038 (mon.0) 236 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T04:14:30.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:29.690159+0000 mon.smithi038 (mon.0) 237 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi038", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T04:14:30.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:29.690993+0000 mon.smithi038 (mon.0) 238 : audit [DBG] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T04:14:30.343 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:30 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:30.024806+0000 mon.smithi038 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.188:0/1749676599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:31.429 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:31.430 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:14:29.685712+0000 mgr.smithi038.bnbogz (mgr.14164) 171 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi038 ... 2023-11-15T04:14:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T04:14:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:14:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:09:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:10:16Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:10:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:14:31.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T04:14:31.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:14:31.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:12:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:12:44Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:13:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:14:31.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:14:31.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:14:31.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: main() 2023-11-15T04:14:31.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:14:31.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: r = ctx.func(ctx) 2023-11-15T04:14:31.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:14:31.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: return func(ctx) 2023-11-15T04:14:31.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T04:14:31.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T04:14:31.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T04:14:31.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T04:14:31.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T04:14:31.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: raise RuntimeError('uid/gid not found') 2023-11-15T04:14:31.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: RuntimeError: uid/gid not found 2023-11-15T04:14:31.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:14:31.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T04:14:31.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: yield (conn, connr) 2023-11-15T04:14:31.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T04:14:31.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: code, '\n'.join(err))) 2023-11-15T04:14:31.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi038 ... 2023-11-15T04:14:31.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T04:14:31.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:14:31.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:09:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:10:16Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:10:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:14:31.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi038 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T04:14:31.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T04:14:31.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:12:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.603 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:12:44Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.603 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr time="2023-11-15T04:13:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T04:14:31.603 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T04:14:31.604 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:14:31.604 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:14:31.604 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: main() 2023-11-15T04:14:31.604 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:14:31.604 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: r = ctx.func(ctx) 2023-11-15T04:14:31.605 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:14:31.605 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: return func(ctx) 2023-11-15T04:14:31.605 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T04:14:31.605 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T04:14:31.605 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T04:14:31.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T04:14:31.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T04:14:31.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: raise RuntimeError('uid/gid not found') 2023-11-15T04:14:31.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: RuntimeError: uid/gid not found 2023-11-15T04:14:31.606 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:14:29.691633+0000 mgr.smithi038.bnbogz (mgr.14164) 172 : cephadm [INF] Deploying daemon crash.smithi038 on smithi038 2023-11-15T04:14:31.607 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:30.098162+0000 mgr.smithi038.bnbogz (mgr.14164) 173 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:32.664 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 -- 172.21.15.188:0/1904353438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 msgr2=0x7f9e280f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:32.665 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 --2- 172.21.15.188:0/1904353438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e280f4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f9e18009fb0 tx=0x7f9e18009b30).stop 2023-11-15T04:14:32.665 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 -- 172.21.15.188:0/1904353438 shutdown_connections 2023-11-15T04:14:32.666 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 --2- 172.21.15.188:0/1904353438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e280f4d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:32.670 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 -- 172.21.15.188:0/1904353438 >> 172.21.15.188:0/1904353438 conn(0x7f9e280f0120 msgr2=0x7f9e280f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:32.671 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 -- 172.21.15.188:0/1904353438 shutdown_connections 2023-11-15T04:14:32.671 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 -- 172.21.15.188:0/1904353438 wait complete. 2023-11-15T04:14:32.671 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.663+0000 7f9e2f191700 1 Processor -- start 2023-11-15T04:14:32.672 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2f191700 1 -- start start 2023-11-15T04:14:32.672 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2f191700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:32.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2f191700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28105890 con 0x7f9e280f4920 2023-11-15T04:14:32.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2e18f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:32.673 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2e18f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57548/0 (socket says 172.21.15.188:57548) 2023-11-15T04:14:32.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2e18f700 1 -- 172.21.15.188:0/3874000324 learned_addr learned my addr 172.21.15.188:0/3874000324 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:32.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.664+0000 7f9e2e18f700 1 -- 172.21.15.188:0/3874000324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18005040 con 0x7f9e280f4920 2023-11-15T04:14:32.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e2e18f700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f9e18013510 tx=0x7f9e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:32.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e18009320 con 0x7f9e280f4920 2023-11-15T04:14:32.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e28105a90 con 0x7f9e280f4920 2023-11-15T04:14:32.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e18013f00 con 0x7f9e280f4920 2023-11-15T04:14:32.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e18014dc0 con 0x7f9e280f4920 2023-11-15T04:14:32.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.665+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e28105f20 con 0x7f9e280f4920 2023-11-15T04:14:32.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e180095f0 con 0x7f9e280f4920 2023-11-15T04:14:32.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28045ad0 con 0x7f9e280f4920 2023-11-15T04:14:32.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e1effd700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 0x7f9e14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:32.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e18041080 con 0x7f9e280f4920 2023-11-15T04:14:32.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e2d98e700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 0x7f9e14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:32.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.666+0000 7f9e2d98e700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 0x7f9e14032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9e240096a0 tx=0x7f9e24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:32.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.671+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e18014420 con 0x7f9e280f4920 2023-11-15T04:14:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.350268+0000 mon.smithi038 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:14:32.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.440644+0000 mon.smithi038 (mon.0) 241 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:14:32.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.443433+0000 mon.smithi038 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:14:32.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.443892+0000 mon.smithi038 (mon.0) 243 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T04:14:32.844 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.449608+0000 mon.smithi038 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:14:32.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.874+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e280fa360 con 0x7f9e280f4920 2023-11-15T04:14:32.877 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.875+0000 7f9e1effd700 1 -- 172.21.15.188:0/3874000324 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e18010230 con 0x7f9e280f4920 2023-11-15T04:14:32.878 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:32.878 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:32.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 msgr2=0x7f9e14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:32.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 0x7f9e14032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9e240096a0 tx=0x7f9e24006b40).stop 2023-11-15T04:14:32.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 msgr2=0x7f9e281053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:32.880 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f9e18013510 tx=0x7f9e1800b040).stop 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 shutdown_connections 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9e1402ff60 0x7f9e14032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 --2- 172.21.15.188:0/3874000324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e280f4920 0x7f9e281053a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 >> 172.21.15.188:0/3874000324 conn(0x7f9e280f0120 msgr2=0x7f9e2805e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 shutdown_connections 2023-11-15T04:14:32.881 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:32.876+0000 7f9e2f191700 1 -- 172.21.15.188:0/3874000324 wait complete. 2023-11-15T04:14:32.882 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:31.444376+0000 mgr.smithi038.bnbogz (mgr.14164) 174 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T04:14:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:14:31.454339+0000 mgr.smithi038.bnbogz (mgr.14164) 175 : cephadm [INF] Deploying daemon grafana.smithi038 on smithi038 2023-11-15T04:14:33.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:32.098478+0000 mgr.smithi038.bnbogz (mgr.14164) 176 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:33.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:32.874980+0000 mon.smithi038 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.188:0/3874000324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:34.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:34.253 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:35.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 -- 172.21.15.188:0/4210519525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 msgr2=0x7f51ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:35.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 --2- 172.21.15.188:0/4210519525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f51e0009fb0 tx=0x7f51e0009b30).stop 2023-11-15T04:14:35.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 -- 172.21.15.188:0/4210519525 shutdown_connections 2023-11-15T04:14:35.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 --2- 172.21.15.188:0/4210519525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:35.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 -- 172.21.15.188:0/4210519525 >> 172.21.15.188:0/4210519525 conn(0x7f51ec0f0b70 msgr2=0x7f51ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:35.532 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 -- 172.21.15.188:0/4210519525 shutdown_connections 2023-11-15T04:14:35.532 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.529+0000 7f51f2be1700 1 -- 172.21.15.188:0/4210519525 wait complete. 2023-11-15T04:14:35.532 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f2be1700 1 Processor -- start 2023-11-15T04:14:35.532 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f2be1700 1 -- start start 2023-11-15T04:14:35.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f2be1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:35.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f2be1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0fb820 con 0x7f51ec0f5410 2023-11-15T04:14:35.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f097d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:35.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f097d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:57576/0 (socket says 172.21.15.188:57576) 2023-11-15T04:14:35.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.530+0000 7f51f097d700 1 -- 172.21.15.188:0/2065929590 learned_addr learned my addr 172.21.15.188:0/2065929590 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:35.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51f097d700 1 -- 172.21.15.188:0/2065929590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0005040 con 0x7f51ec0f5410 2023-11-15T04:14:35.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51f097d700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f51e0013510 tx=0x7f51e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:35.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e0009320 con 0x7f51ec0f5410 2023-11-15T04:14:35.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec0fb9c0 con 0x7f51ec0f5410 2023-11-15T04:14:35.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51e0013f00 con 0x7f51ec0f5410 2023-11-15T04:14:35.535 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e0014d90 con 0x7f51ec0f5410 2023-11-15T04:14:35.535 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.531+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec0fbe50 con 0x7f51ec0f5410 2023-11-15T04:14:35.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.532+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51e00095f0 con 0x7f51ec0f5410 2023-11-15T04:14:35.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.532+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec059100 con 0x7f51ec0f5410 2023-11-15T04:14:35.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.532+0000 7f51e97fa700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 0x7f51dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:35.537 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.532+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51e0040fc0 con 0x7f51ec0f5410 2023-11-15T04:14:35.537 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.532+0000 7f51ebfff700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 0x7f51dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:35.537 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.533+0000 7f51ebfff700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 0x7f51dc032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f51d80096a0 tx=0x7f51d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:35.538 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.537+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51e0014420 con 0x7f51ec0f5410 2023-11-15T04:14:35.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.741+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51ec045ad0 con 0x7f51ec0f5410 2023-11-15T04:14:35.742 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.742+0000 7f51e97fa700 1 -- 172.21.15.188:0/2065929590 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51e0010230 con 0x7f51ec0f5410 2023-11-15T04:14:35.743 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:35.743 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:35.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 msgr2=0x7f51dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:35.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 0x7f51dc032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f51d80096a0 tx=0x7f51d8006b40).stop 2023-11-15T04:14:35.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 msgr2=0x7f51ec0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:35.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f51e0013510 tx=0x7f51e000b040).stop 2023-11-15T04:14:35.744 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 shutdown_connections 2023-11-15T04:14:35.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f51dc02ff60 0x7f51dc032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:35.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 --2- 172.21.15.188:0/2065929590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec0f5410 0x7f51ec0fd3a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:35.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 >> 172.21.15.188:0/2065929590 conn(0x7f51ec0f0b70 msgr2=0x7f51ec05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:35.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 shutdown_connections 2023-11-15T04:14:35.745 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:35.744+0000 7f51f2be1700 1 -- 172.21.15.188:0/2065929590 wait complete. 2023-11-15T04:14:35.746 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:35.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:34.098837+0000 mgr.smithi038.bnbogz (mgr.14164) 177 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:36.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:36 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:35.742515+0000 mon.smithi038 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.188:0/2065929590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:37.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:37.150 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:37.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:36.099232+0000 mgr.smithi038.bnbogz (mgr.14164) 178 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:38.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 -- 172.21.15.188:0/600278015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac0f6510 msgr2=0x7f67ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:38.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/600278015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac0f6510 0x7f67ac0f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f67a0009fb0 tx=0x7f67a0009b30).stop 2023-11-15T04:14:38.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 -- 172.21.15.188:0/600278015 shutdown_connections 2023-11-15T04:14:38.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/600278015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac0f6510 0x7f67ac0f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:38.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 -- 172.21.15.188:0/600278015 >> 172.21.15.188:0/600278015 conn(0x7f67ac0f1c70 msgr2=0x7f67ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:38.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 -- 172.21.15.188:0/600278015 shutdown_connections 2023-11-15T04:14:38.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.460+0000 7f67b2ee9700 1 -- 172.21.15.188:0/600278015 wait complete. 2023-11-15T04:14:38.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b2ee9700 1 Processor -- start 2023-11-15T04:14:38.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b2ee9700 1 -- start start 2023-11-15T04:14:38.463 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b2ee9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:38.464 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b2ee9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac107150 con 0x7f67ac106840 2023-11-15T04:14:38.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b0c85700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:38.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b0c85700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59512/0 (socket says 172.21.15.188:59512) 2023-11-15T04:14:38.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.461+0000 7f67b0c85700 1 -- 172.21.15.188:0/2523432157 learned_addr learned my addr 172.21.15.188:0/2523432157 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:38.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67b0c85700 1 -- 172.21.15.188:0/2523432157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a0005040 con 0x7f67ac106840 2023-11-15T04:14:38.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67b0c85700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f67a0009f80 tx=0x7f67a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:38.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67a00136c0 con 0x7f67ac106840 2023-11-15T04:14:38.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac107350 con 0x7f67ac106840 2023-11-15T04:14:38.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67a0013e70 con 0x7f67ac106840 2023-11-15T04:14:38.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67a0014420 con 0x7f67ac106840 2023-11-15T04:14:38.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.462+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac107ea0 con 0x7f67ac106840 2023-11-15T04:14:38.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67a0025060 con 0x7f67ac106840 2023-11-15T04:14:38.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67a97fa700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 0x7f679c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:38.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67a0041230 con 0x7f67ac106840 2023-11-15T04:14:38.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac059100 con 0x7f67ac106840 2023-11-15T04:14:38.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67abfff700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 0x7f679c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:38.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.463+0000 7f67abfff700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 0x7f679c032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f67980096a0 tx=0x7f6798006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:38.470 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.468+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67a0028440 con 0x7f67ac106840 2023-11-15T04:14:38.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.673+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67ac1085a0 con 0x7f67ac106840 2023-11-15T04:14:38.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.674+0000 7f67a97fa700 1 -- 172.21.15.188:0/2523432157 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f67a0026340 con 0x7f67ac106840 2023-11-15T04:14:38.675 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:38.675 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:38.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.675+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 msgr2=0x7f679c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:38.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.675+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 0x7f679c032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f67980096a0 tx=0x7f6798006b40).stop 2023-11-15T04:14:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.675+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 msgr2=0x7f67ac106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.675+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f67a0009f80 tx=0x7f67a0009b30).stop 2023-11-15T04:14:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 shutdown_connections 2023-11-15T04:14:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f679c02ff60 0x7f679c032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:38.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 --2- 172.21.15.188:0/2523432157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67ac106840 0x7f67ac106c60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:38.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 >> 172.21.15.188:0/2523432157 conn(0x7f67ac0f1c70 msgr2=0x7f67ac0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:38.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 shutdown_connections 2023-11-15T04:14:38.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:38.676+0000 7f67b2ee9700 1 -- 172.21.15.188:0/2523432157 wait complete. 2023-11-15T04:14:38.678 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:39.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:38.099616+0000 mgr.smithi038.bnbogz (mgr.14164) 179 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:39.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:38.675132+0000 mon.smithi038 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.188:0/2523432157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:40.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:40.068 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:41.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 -- 172.21.15.188:0/1599016902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 msgr2=0x7fa5e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:41.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/1599016902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e40f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fa5d4009fb0 tx=0x7fa5d4009b30).stop 2023-11-15T04:14:41.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 -- 172.21.15.188:0/1599016902 shutdown_connections 2023-11-15T04:14:41.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/1599016902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e40f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:41.318 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 -- 172.21.15.188:0/1599016902 >> 172.21.15.188:0/1599016902 conn(0x7fa5e40f0b70 msgr2=0x7fa5e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:41.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 -- 172.21.15.188:0/1599016902 shutdown_connections 2023-11-15T04:14:41.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.315+0000 7fa5ebda3700 1 -- 172.21.15.188:0/1599016902 wait complete. 2023-11-15T04:14:41.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5ebda3700 1 Processor -- start 2023-11-15T04:14:41.319 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5ebda3700 1 -- start start 2023-11-15T04:14:41.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5ebda3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:41.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5ebda3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e4060220 con 0x7fa5e40f5410 2023-11-15T04:14:41.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5e9b3f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:41.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5e9b3f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59534/0 (socket says 172.21.15.188:59534) 2023-11-15T04:14:41.320 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5e9b3f700 1 -- 172.21.15.188:0/4074596695 learned_addr learned my addr 172.21.15.188:0/4074596695 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:41.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5e9b3f700 1 -- 172.21.15.188:0/4074596695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4005040 con 0x7fa5e40f5410 2023-11-15T04:14:41.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.316+0000 7fa5e9b3f700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fa5d4013510 tx=0x7fa5d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:41.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5d4009320 con 0x7fa5e40f5410 2023-11-15T04:14:41.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e4060420 con 0x7fa5e40f5410 2023-11-15T04:14:41.321 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5d4013f00 con 0x7fa5e40f5410 2023-11-15T04:14:41.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5d4014d90 con 0x7fa5e40f5410 2023-11-15T04:14:41.322 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e4109b00 con 0x7fa5e40f5410 2023-11-15T04:14:41.323 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.317+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5d40095f0 con 0x7fa5e40f5410 2023-11-15T04:14:41.323 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.318+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e4059100 con 0x7fa5e40f5410 2023-11-15T04:14:41.323 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.318+0000 7fa5da7fc700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 0x7fa5d0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:41.323 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.318+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5d4041080 con 0x7fa5e40f5410 2023-11-15T04:14:41.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.318+0000 7fa5e933e700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 0x7fa5d0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:41.324 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.318+0000 7fa5e933e700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 0x7fa5d0032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa5e00096a0 tx=0x7fa5e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:41.326 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.326+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5d406d2b0 con 0x7fa5e40f5410 2023-11-15T04:14:41.530 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.529+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5e4045ad0 con 0x7fa5e40f5410 2023-11-15T04:14:41.531 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.529+0000 7fa5da7fc700 1 -- 172.21.15.188:0/4074596695 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5d4014420 con 0x7fa5e40f5410 2023-11-15T04:14:41.531 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:41.532 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:41.533 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 msgr2=0x7fa5d0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:41.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 0x7fa5d0032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa5e00096a0 tx=0x7fa5e0006b40).stop 2023-11-15T04:14:41.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 msgr2=0x7fa5e4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:41.534 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fa5d4013510 tx=0x7fa5d400b040).stop 2023-11-15T04:14:41.535 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 shutdown_connections 2023-11-15T04:14:41.535 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa5d002ffb0 0x7fa5d0032470 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:41.535 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 --2- 172.21.15.188:0/4074596695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa5e40f5410 0x7fa5e4109450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:41.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 >> 172.21.15.188:0/4074596695 conn(0x7fa5e40f0b70 msgr2=0x7fa5e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:41.536 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 shutdown_connections 2023-11-15T04:14:41.537 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:41.531+0000 7fa5ebda3700 1 -- 172.21.15.188:0/4074596695 wait complete. 2023-11-15T04:14:41.537 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:41.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:40.099872+0000 mgr.smithi038.bnbogz (mgr.14164) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:41.530835+0000 mon.smithi038 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.188:0/4074596695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:42.100101+0000 mgr.smithi038.bnbogz (mgr.14164) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:42.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:42.928 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:44.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- 172.21.15.188:0/3332112888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 msgr2=0x7fbaec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:44.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 --2- 172.21.15.188:0/3332112888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec0f66c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fbadc009fb0 tx=0x7fbadc009b30).stop 2023-11-15T04:14:44.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- 172.21.15.188:0/3332112888 shutdown_connections 2023-11-15T04:14:44.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 --2- 172.21.15.188:0/3332112888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec0f66c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:44.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- 172.21.15.188:0/3332112888 >> 172.21.15.188:0/3332112888 conn(0x7fbaec0f1aa0 msgr2=0x7fbaec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:44.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- 172.21.15.188:0/3332112888 shutdown_connections 2023-11-15T04:14:44.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- 172.21.15.188:0/3332112888 wait complete. 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 Processor -- start 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.233+0000 7fbaebfff700 1 -- start start 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaebfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaebfff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec069e20 con 0x7fbaec0f62a0 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:44.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59558/0 (socket says 172.21.15.188:59558) 2023-11-15T04:14:44.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaeaffd700 1 -- 172.21.15.188:0/4280757044 learned_addr learned my addr 172.21.15.188:0/4280757044 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:44.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaeaffd700 1 -- 172.21.15.188:0/4280757044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc005040 con 0x7fbaec0f62a0 2023-11-15T04:14:44.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaeaffd700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fbadc013510 tx=0x7fbadc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:44.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc009320 con 0x7fbaec0f62a0 2023-11-15T04:14:44.239 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbadc013f00 con 0x7fbaec0f62a0 2023-11-15T04:14:44.239 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc014dc0 con 0x7fbaec0f62a0 2023-11-15T04:14:44.239 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec069fc0 con 0x7fbaec0f62a0 2023-11-15T04:14:44.239 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.234+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec06a450 con 0x7fbaec0f62a0 2023-11-15T04:14:44.240 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.235+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbadc0263a0 con 0x7fbaec0f62a0 2023-11-15T04:14:44.241 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.235+0000 7fbad37fe700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 0x7fbad4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:44.241 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.235+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbadc0410b0 con 0x7fbaec0f62a0 2023-11-15T04:14:44.241 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.235+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec059100 con 0x7fbaec0f62a0 2023-11-15T04:14:44.241 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.235+0000 7fbaea7fc700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 0x7fbad4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:44.241 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.236+0000 7fbaea7fc700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 0x7fbad4032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbae00096a0 tx=0x7fbae0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:44.242 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.241+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbadc012920 con 0x7fbaec0f62a0 2023-11-15T04:14:44.446 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.445+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbaec0fbce0 con 0x7fbaec0f62a0 2023-11-15T04:14:44.447 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.446+0000 7fbad37fe700 1 -- 172.21.15.188:0/4280757044 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbadc02b020 con 0x7fbaec0f62a0 2023-11-15T04:14:44.447 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:44.448 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:44.448 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 msgr2=0x7fbad4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:44.449 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 0x7fbad4032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbae00096a0 tx=0x7fbae0006b40).stop 2023-11-15T04:14:44.449 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 msgr2=0x7fbaec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:44.449 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fbadc013510 tx=0x7fbadc00b040).stop 2023-11-15T04:14:44.449 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 shutdown_connections 2023-11-15T04:14:44.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fbad402ffb0 0x7fbad4032470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:44.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 --2- 172.21.15.188:0/4280757044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaec0f62a0 0x7fbaec06b9a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:44.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 >> 172.21.15.188:0/4280757044 conn(0x7fbaec0f1aa0 msgr2=0x7fbaec0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:44.450 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 shutdown_connections 2023-11-15T04:14:44.451 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:44.448+0000 7fbaebfff700 1 -- 172.21.15.188:0/4280757044 wait complete. 2023-11-15T04:14:44.451 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:44.100367+0000 mgr.smithi038.bnbogz (mgr.14164) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:44.447522+0000 mon.smithi038 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.188:0/4280757044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:45.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:45.839 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:47.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.083+0000 7f69d07b2700 1 -- 172.21.15.188:0/1004229665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 msgr2=0x7f69c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:47.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.083+0000 7f69d07b2700 1 --2- 172.21.15.188:0/1004229665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c80f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f69bc009fb0 tx=0x7f69bc009b30).stop 2023-11-15T04:14:47.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- 172.21.15.188:0/1004229665 shutdown_connections 2023-11-15T04:14:47.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 --2- 172.21.15.188:0/1004229665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c80f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:47.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- 172.21.15.188:0/1004229665 >> 172.21.15.188:0/1004229665 conn(0x7f69c80f0b70 msgr2=0x7f69c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:47.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- 172.21.15.188:0/1004229665 shutdown_connections 2023-11-15T04:14:47.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- 172.21.15.188:0/1004229665 wait complete. 2023-11-15T04:14:47.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 Processor -- start 2023-11-15T04:14:47.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- start start 2023-11-15T04:14:47.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:47.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.084+0000 7f69d07b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8060220 con 0x7f69c80f5410 2023-11-15T04:14:47.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69ce54e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:47.088 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69ce54e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59572/0 (socket says 172.21.15.188:59572) 2023-11-15T04:14:47.088 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69ce54e700 1 -- 172.21.15.188:0/3566490 learned_addr learned my addr 172.21.15.188:0/3566490 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:47.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69ce54e700 1 -- 172.21.15.188:0/3566490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc005040 con 0x7f69c80f5410 2023-11-15T04:14:47.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69ce54e700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f69bc013510 tx=0x7f69bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:47.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc009320 con 0x7f69c80f5410 2023-11-15T04:14:47.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c8060420 con 0x7f69c80f5410 2023-11-15T04:14:47.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c8109b00 con 0x7f69c80f5410 2023-11-15T04:14:47.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69bc013f00 con 0x7f69c80f5410 2023-11-15T04:14:47.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.085+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc014d60 con 0x7f69c80f5410 2023-11-15T04:14:47.091 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.086+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69bc0095f0 con 0x7f69c80f5410 2023-11-15T04:14:47.091 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.086+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8059100 con 0x7f69c80f5410 2023-11-15T04:14:47.091 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.086+0000 7f69baffd700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 0x7f69b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:47.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.086+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69bc040ff0 con 0x7f69c80f5410 2023-11-15T04:14:47.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.086+0000 7f69cdd4d700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 0x7f69b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:47.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.087+0000 7f69cdd4d700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 0x7f69b4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:47.092 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.091+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69bc014420 con 0x7f69c80f5410 2023-11-15T04:14:47.299 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.298+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69c8045ad0 con 0x7f69c80f5410 2023-11-15T04:14:47.299 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.299+0000 7f69baffd700 1 -- 172.21.15.188:0/3566490 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69bc012980 con 0x7f69c80f5410 2023-11-15T04:14:47.300 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:47.300 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:47.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 msgr2=0x7f69b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:47.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 0x7f69b4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).stop 2023-11-15T04:14:47.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 msgr2=0x7f69c8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:47.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f69bc013510 tx=0x7f69bc00b040).stop 2023-11-15T04:14:47.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 shutdown_connections 2023-11-15T04:14:47.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f69b402ff60 0x7f69b4032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:47.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 --2- 172.21.15.188:0/3566490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c80f5410 0x7f69c8109450 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:47.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 >> 172.21.15.188:0/3566490 conn(0x7f69c80f0b70 msgr2=0x7f69c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:47.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 shutdown_connections 2023-11-15T04:14:47.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:47.300+0000 7f69d07b2700 1 -- 172.21.15.188:0/3566490 wait complete. 2023-11-15T04:14:47.303 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:47.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:46.100652+0000 mgr.smithi038.bnbogz (mgr.14164) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:48.414 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:48 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:47.300056+0000 mon.smithi038 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.188:0/3566490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:48.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:48.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:49.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:48.100944+0000 mgr.smithi038.bnbogz (mgr.14164) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:49.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:48.242659+0000 mon.smithi038 (mon.0) 251 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' 2023-11-15T04:14:49.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:14:48.244230+0000 mgr.smithi038.bnbogz (mgr.14164) 185 : cephadm [INF] Deploying daemon node-exporter.smithi038 on smithi038 2023-11-15T04:14:49.935 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 -- 172.21.15.188:0/4196049188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 msgr2=0x7f50f00f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:49.936 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 --2- 172.21.15.188:0/4196049188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f00f9130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f50e0009fb0 tx=0x7f50e0009b30).stop 2023-11-15T04:14:49.936 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 -- 172.21.15.188:0/4196049188 shutdown_connections 2023-11-15T04:14:49.936 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 --2- 172.21.15.188:0/4196049188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f00f9130 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:49.936 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 -- 172.21.15.188:0/4196049188 >> 172.21.15.188:0/4196049188 conn(0x7f50f00f0b70 msgr2=0x7f50f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:49.937 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.933+0000 7f50f7455700 1 -- 172.21.15.188:0/4196049188 shutdown_connections 2023-11-15T04:14:49.937 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f7455700 1 -- 172.21.15.188:0/4196049188 wait complete. 2023-11-15T04:14:49.937 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f7455700 1 Processor -- start 2023-11-15T04:14:49.937 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f7455700 1 -- start start 2023-11-15T04:14:49.938 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f7455700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:49.938 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f7455700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0106bb0 con 0x7f50f0060240 2023-11-15T04:14:49.938 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f51f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:49.938 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f51f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59052/0 (socket says 172.21.15.188:59052) 2023-11-15T04:14:49.938 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.934+0000 7f50f51f1700 1 -- 172.21.15.188:0/3388049928 learned_addr learned my addr 172.21.15.188:0/3388049928 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:49.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50f51f1700 1 -- 172.21.15.188:0/3388049928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e0005040 con 0x7f50f0060240 2023-11-15T04:14:49.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50f51f1700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f50e0013510 tx=0x7f50e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:49.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e0009320 con 0x7f50f0060240 2023-11-15T04:14:49.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f0106db0 con 0x7f50f0060240 2023-11-15T04:14:49.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50e0013f00 con 0x7f50f0060240 2023-11-15T04:14:49.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e0014dc0 con 0x7f50f0060240 2023-11-15T04:14:49.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.935+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f0107240 con 0x7f50f0060240 2023-11-15T04:14:49.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.936+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f0045ad0 con 0x7f50f0060240 2023-11-15T04:14:49.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.936+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50e00095f0 con 0x7f50f0060240 2023-11-15T04:14:49.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.936+0000 7f50e5ffb700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 0x7f50dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:49.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.936+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50e0041020 con 0x7f50f0060240 2023-11-15T04:14:49.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.936+0000 7f50f49f0700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 0x7f50dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:49.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.937+0000 7f50f49f0700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 0x7f50dc032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f50ec0096a0 tx=0x7f50ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:49.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:49.941+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50e0014420 con 0x7f50f0060240 2023-11-15T04:14:50.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.144+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f0108090 con 0x7f50f0060240 2023-11-15T04:14:50.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.145+0000 7f50e5ffb700 1 -- 172.21.15.188:0/3388049928 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50e001c920 con 0x7f50f0060240 2023-11-15T04:14:50.146 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:50.146 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:50.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 msgr2=0x7f50dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:50.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 0x7f50dc032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f50ec0096a0 tx=0x7f50ec006b40).stop 2023-11-15T04:14:50.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 msgr2=0x7f50f01066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:50.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f50e0013510 tx=0x7f50e000b040).stop 2023-11-15T04:14:50.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 shutdown_connections 2023-11-15T04:14:50.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f50dc02ff60 0x7f50dc032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:50.148 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 --2- 172.21.15.188:0/3388049928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0060240 0x7f50f01066c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:50.149 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 >> 172.21.15.188:0/3388049928 conn(0x7f50f00f0b70 msgr2=0x7f50f00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:50.149 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.146+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 shutdown_connections 2023-11-15T04:14:50.149 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:50.147+0000 7f50f7455700 1 -- 172.21.15.188:0/3388049928 wait complete. 2023-11-15T04:14:50.149 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:50.146257+0000 mon.smithi038 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.188:0/3388049928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:50.212788+0000 mon.smithi038 (mon.0) 253 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:14:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:50.214792+0000 mon.smithi038 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:14:51.542 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:51.542 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:50.101481+0000 mgr.smithi038.bnbogz (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:52.747 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 -- 172.21.15.188:0/677770048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 msgr2=0x7f36480f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:52.747 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 --2- 172.21.15.188:0/677770048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f36480f4be0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f3638009fb0 tx=0x7f3638009b30).stop 2023-11-15T04:14:52.747 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 -- 172.21.15.188:0/677770048 shutdown_connections 2023-11-15T04:14:52.747 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 --2- 172.21.15.188:0/677770048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f36480f4be0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:52.748 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 -- 172.21.15.188:0/677770048 >> 172.21.15.188:0/677770048 conn(0x7f36480f02e0 msgr2=0x7f36480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:52.748 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 -- 172.21.15.188:0/677770048 shutdown_connections 2023-11-15T04:14:52.748 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.744+0000 7f364d094700 1 -- 172.21.15.188:0/677770048 wait complete. 2023-11-15T04:14:52.748 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f364d094700 1 Processor -- start 2023-11-15T04:14:52.748 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f364d094700 1 -- start start 2023-11-15T04:14:52.749 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f364d094700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:52.749 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f364d094700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3648106bf0 con 0x7f36480f47c0 2023-11-15T04:14:52.749 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f3646d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:52.749 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f3646d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59066/0 (socket says 172.21.15.188:59066) 2023-11-15T04:14:52.749 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.745+0000 7f3646d9d700 1 -- 172.21.15.188:0/2489155390 learned_addr learned my addr 172.21.15.188:0/2489155390 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:52.750 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f3646d9d700 1 -- 172.21.15.188:0/2489155390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3638005040 con 0x7f36480f47c0 2023-11-15T04:14:52.750 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f3646d9d700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3638013510 tx=0x7f363800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:52.750 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3638009320 con 0x7f36480f47c0 2023-11-15T04:14:52.750 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3638013f00 con 0x7f36480f47c0 2023-11-15T04:14:52.750 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3638014d90 con 0x7f36480f47c0 2023-11-15T04:14:52.751 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3648106df0 con 0x7f36480f47c0 2023-11-15T04:14:52.751 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.746+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3648107280 con 0x7f36480f47c0 2023-11-15T04:14:52.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.747+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36380263a0 con 0x7f36480f47c0 2023-11-15T04:14:52.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.747+0000 7f362f7fe700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 0x7f3630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:52.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.747+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3638041090 con 0x7f36480f47c0 2023-11-15T04:14:52.752 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.747+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3648045ad0 con 0x7f36480f47c0 2023-11-15T04:14:52.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.747+0000 7f364659c700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 0x7f3630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:52.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.748+0000 7f364659c700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 0x7f3630032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f363c0096a0 tx=0x7f363c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:52.753 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.752+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3638012920 con 0x7f36480f47c0 2023-11-15T04:14:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:52.101805+0000 mgr.smithi038.bnbogz (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:52.955 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.954+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36480fc0b0 con 0x7f36480f47c0 2023-11-15T04:14:52.956 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.955+0000 7f362f7fe700 1 -- 172.21.15.188:0/2489155390 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f363802b020 con 0x7f36480f47c0 2023-11-15T04:14:52.956 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:52.956 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:52.957 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 msgr2=0x7f3630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:52.957 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 0x7f3630032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f363c0096a0 tx=0x7f363c006b40).stop 2023-11-15T04:14:52.958 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 msgr2=0x7f3648106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:52.958 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3638013510 tx=0x7f363800b040).stop 2023-11-15T04:14:52.958 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 shutdown_connections 2023-11-15T04:14:52.958 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f363002ff60 0x7f3630032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:52.958 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 --2- 172.21.15.188:0/2489155390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36480f47c0 0x7f3648106700 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:52.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.956+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 >> 172.21.15.188:0/2489155390 conn(0x7f36480f02e0 msgr2=0x7f36480f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:52.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.957+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 shutdown_connections 2023-11-15T04:14:52.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:52.957+0000 7f364d094700 1 -- 172.21.15.188:0/2489155390 wait complete. 2023-11-15T04:14:52.960 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:53.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:52.956432+0000 mon.smithi038 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.188:0/2489155390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:54.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:54.398 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:54.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:54.102057+0000 mgr.smithi038.bnbogz (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:55.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- 172.21.15.188:0/1805417685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 msgr2=0x7f680c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:55.699 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 --2- 172.21.15.188:0/1805417685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c0f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f67fc009fb0 tx=0x7f67fc009b30).stop 2023-11-15T04:14:55.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- 172.21.15.188:0/1805417685 shutdown_connections 2023-11-15T04:14:55.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 --2- 172.21.15.188:0/1805417685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c0f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:55.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- 172.21.15.188:0/1805417685 >> 172.21.15.188:0/1805417685 conn(0x7f680c0f1c70 msgr2=0x7f680c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:55.700 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- 172.21.15.188:0/1805417685 shutdown_connections 2023-11-15T04:14:55.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- 172.21.15.188:0/1805417685 wait complete. 2023-11-15T04:14:55.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 Processor -- start 2023-11-15T04:14:55.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.698+0000 7f6811c18700 1 -- start start 2023-11-15T04:14:55.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f6811c18700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:55.701 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f6811c18700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c069e20 con 0x7f680c0f6510 2023-11-15T04:14:55.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f680b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:55.703 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f680b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59082/0 (socket says 172.21.15.188:59082) 2023-11-15T04:14:55.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f680b7fe700 1 -- 172.21.15.188:0/3802309022 learned_addr learned my addr 172.21.15.188:0/3802309022 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:55.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f680b7fe700 1 -- 172.21.15.188:0/3802309022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc005040 con 0x7f680c0f6510 2023-11-15T04:14:55.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.699+0000 7f680b7fe700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f67fc013510 tx=0x7f67fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:55.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67fc009320 con 0x7f680c0f6510 2023-11-15T04:14:55.704 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c069fc0 con 0x7f680c0f6510 2023-11-15T04:14:55.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c06a450 con 0x7f680c0f6510 2023-11-15T04:14:55.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67fc013f00 con 0x7f680c0f6510 2023-11-15T04:14:55.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67fc014d60 con 0x7f680c0f6510 2023-11-15T04:14:55.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67fc0095f0 con 0x7f680c0f6510 2023-11-15T04:14:55.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.700+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c059100 con 0x7f680c0f6510 2023-11-15T04:14:55.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.701+0000 7f67f3fff700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 0x7f67f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:55.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.701+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67fc0410d0 con 0x7f680c0f6510 2023-11-15T04:14:55.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.701+0000 7f680affd700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 0x7f67f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:55.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.701+0000 7f680affd700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 0x7f67f4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f68000096a0 tx=0x7f6800006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:55.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.706+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67fc014420 con 0x7f680c0f6510 2023-11-15T04:14:55.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.908+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f680c0fbf60 con 0x7f680c0f6510 2023-11-15T04:14:55.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.909+0000 7f67f3fff700 1 -- 172.21.15.188:0/3802309022 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f67fc012980 con 0x7f680c0f6510 2023-11-15T04:14:55.909 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:55.910 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:55.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 msgr2=0x7f67f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:55.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 0x7f67f4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f68000096a0 tx=0x7f6800006b40).stop 2023-11-15T04:14:55.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 msgr2=0x7f680c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:55.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f67fc013510 tx=0x7f67fc00b040).stop 2023-11-15T04:14:55.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 shutdown_connections 2023-11-15T04:14:55.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f67f402ff60 0x7f67f4032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:55.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 --2- 172.21.15.188:0/3802309022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f680c0f6510 0x7f680c06b9a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:55.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 >> 172.21.15.188:0/3802309022 conn(0x7f680c0f1c70 msgr2=0x7f680c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:55.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 shutdown_connections 2023-11-15T04:14:55.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:55.910+0000 7f6811c18700 1 -- 172.21.15.188:0/3802309022 wait complete. 2023-11-15T04:14:55.913 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:56.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:55.909967+0000 mon.smithi038 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.188:0/3802309022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:14:57.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:14:57.296 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:14:57.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:56 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:56.102421+0000 mgr.smithi038.bnbogz (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:58.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 -- 172.21.15.188:0/2732883102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 msgr2=0x7fe5200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:58.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 --2- 172.21.15.188:0/2732883102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe5200f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fe514009fb0 tx=0x7fe514009b30).stop 2023-11-15T04:14:58.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 -- 172.21.15.188:0/2732883102 shutdown_connections 2023-11-15T04:14:58.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 --2- 172.21.15.188:0/2732883102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe5200f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:58.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 -- 172.21.15.188:0/2732883102 >> 172.21.15.188:0/2732883102 conn(0x7fe5200f0b70 msgr2=0x7fe5200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:58.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 -- 172.21.15.188:0/2732883102 shutdown_connections 2023-11-15T04:14:58.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.556+0000 7fe52708d700 1 -- 172.21.15.188:0/2732883102 wait complete. 2023-11-15T04:14:58.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.557+0000 7fe52708d700 1 Processor -- start 2023-11-15T04:14:58.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.557+0000 7fe52708d700 1 -- start start 2023-11-15T04:14:58.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.557+0000 7fe52708d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:58.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.557+0000 7fe52708d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe520105b20 con 0x7fe5200f5410 2023-11-15T04:14:58.561 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.557+0000 7fe524e29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:58.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe524e29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51626/0 (socket says 172.21.15.188:51626) 2023-11-15T04:14:58.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe524e29700 1 -- 172.21.15.188:0/3309821819 learned_addr learned my addr 172.21.15.188:0/3309821819 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:14:58.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe524e29700 1 -- 172.21.15.188:0/3309821819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe514005040 con 0x7fe5200f5410 2023-11-15T04:14:58.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe524e29700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe514013510 tx=0x7fe51400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:58.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe514009320 con 0x7fe5200f5410 2023-11-15T04:14:58.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe520105d20 con 0x7fe5200f5410 2023-11-15T04:14:58.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe514013f00 con 0x7fe5200f5410 2023-11-15T04:14:58.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5201061b0 con 0x7fe5200f5410 2023-11-15T04:14:58.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.558+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe514014d90 con 0x7fe5200f5410 2023-11-15T04:14:58.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.559+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe514014420 con 0x7fe5200f5410 2023-11-15T04:14:58.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.559+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe520045ad0 con 0x7fe5200f5410 2023-11-15T04:14:58.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.559+0000 7fe51d7fa700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 0x7fe510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:14:58.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.559+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe514041100 con 0x7fe5200f5410 2023-11-15T04:14:58.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.559+0000 7fe51ffff700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 0x7fe510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:14:58.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.560+0000 7fe51ffff700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 0x7fe510032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe50c0096a0 tx=0x7fe50c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:14:58.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.564+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5140095f0 con 0x7fe5200f5410 2023-11-15T04:14:58.769 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.769+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5200fae60 con 0x7fe5200f5410 2023-11-15T04:14:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.770+0000 7fe51d7fa700 1 -- 172.21.15.188:0/3309821819 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe51401db70 con 0x7fe5200f5410 2023-11-15T04:14:58.770 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:14:58.771 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:14:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 msgr2=0x7fe510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 0x7fe510032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe50c0096a0 tx=0x7fe50c006b40).stop 2023-11-15T04:14:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 msgr2=0x7fe520105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:14:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe514013510 tx=0x7fe51400b040).stop 2023-11-15T04:14:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 shutdown_connections 2023-11-15T04:14:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe51002ff60 0x7fe510032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 --2- 172.21.15.188:0/3309821819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5200f5410 0x7fe520105630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:14:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 >> 172.21.15.188:0/3309821819 conn(0x7fe5200f0b70 msgr2=0x7fe5200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:14:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 shutdown_connections 2023-11-15T04:14:58.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:14:58.771+0000 7fe52708d700 1 -- 172.21.15.188:0/3309821819 wait complete. 2023-11-15T04:14:58.774 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:14:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:14:58.102733+0000 mgr.smithi038.bnbogz (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:14:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:14:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:14:58.770897+0000 mon.smithi038 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.188:0/3309821819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:00.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:00.204 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:01.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- 172.21.15.188:0/85515731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 msgr2=0x7f1e280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:01.482 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 --2- 172.21.15.188:0/85515731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e280f55c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f1e18009fb0 tx=0x7f1e18009b30).stop 2023-11-15T04:15:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- 172.21.15.188:0/85515731 shutdown_connections 2023-11-15T04:15:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 --2- 172.21.15.188:0/85515731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e280f55c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- 172.21.15.188:0/85515731 >> 172.21.15.188:0/85515731 conn(0x7f1e280f09a0 msgr2=0x7f1e280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- 172.21.15.188:0/85515731 shutdown_connections 2023-11-15T04:15:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- 172.21.15.188:0/85515731 wait complete. 2023-11-15T04:15:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 Processor -- start 2023-11-15T04:15:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.481+0000 7f1e2759e700 1 -- start start 2023-11-15T04:15:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2759e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2759e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28060220 con 0x7f1e280f51a0 2023-11-15T04:15:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51646/0 (socket says 172.21.15.188:51646) 2023-11-15T04:15:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2659c700 1 -- 172.21.15.188:0/1973420595 learned_addr learned my addr 172.21.15.188:0/1973420595 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.482+0000 7f1e2659c700 1 -- 172.21.15.188:0/1973420595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18005040 con 0x7f1e280f51a0 2023-11-15T04:15:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e2659c700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1e18013510 tx=0x7f1e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e18009320 con 0x7f1e280f51a0 2023-11-15T04:15:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e28060420 con 0x7f1e280f51a0 2023-11-15T04:15:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e18013f00 con 0x7f1e280f51a0 2023-11-15T04:15:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e18014dc0 con 0x7f1e280f51a0 2023-11-15T04:15:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.483+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e28109880 con 0x7f1e280f51a0 2023-11-15T04:15:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.484+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e1801d3d0 con 0x7f1e280f51a0 2023-11-15T04:15:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.484+0000 7f1e16ffd700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 0x7f1e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.484+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28059100 con 0x7f1e280f51a0 2023-11-15T04:15:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.484+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e18041150 con 0x7f1e280f51a0 2023-11-15T04:15:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.484+0000 7f1e25d9b700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 0x7f1e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.485+0000 7f1e25d9b700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 0x7f1e10032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1e1c0096a0 tx=0x7f1e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.489+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e18012510 con 0x7f1e280f51a0 2023-11-15T04:15:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:00.103029+0000 mgr.smithi038.bnbogz (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:01.694 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.693+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e280fabe0 con 0x7f1e280f51a0 2023-11-15T04:15:01.695 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.694+0000 7f1e16ffd700 1 -- 172.21.15.188:0/1973420595 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e18012330 con 0x7f1e280f51a0 2023-11-15T04:15:01.695 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:01.695 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:01.696 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 msgr2=0x7f1e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 0x7f1e10032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1e1c0096a0 tx=0x7f1e1c006b40).stop 2023-11-15T04:15:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 msgr2=0x7f1e281091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1e18013510 tx=0x7f1e1800b040).stop 2023-11-15T04:15:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 shutdown_connections 2023-11-15T04:15:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e1002ff60 0x7f1e10032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 --2- 172.21.15.188:0/1973420595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e280f51a0 0x7f1e281091d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 >> 172.21.15.188:0/1973420595 conn(0x7f1e280f09a0 msgr2=0x7f1e280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 shutdown_connections 2023-11-15T04:15:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:01.696+0000 7f1e2759e700 1 -- 172.21.15.188:0/1973420595 wait complete. 2023-11-15T04:15:01.699 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:01.695650+0000 mon.smithi038 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.188:0/1973420595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:02.843 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:02.103302+0000 mgr.smithi038.bnbogz (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:03.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:03.081 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:04.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- 172.21.15.188:0/663857715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 msgr2=0x7f64f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:04.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 --2- 172.21.15.188:0/663857715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f00f55c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f64e4009fb0 tx=0x7f64e4009b30).stop 2023-11-15T04:15:04.413 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- 172.21.15.188:0/663857715 shutdown_connections 2023-11-15T04:15:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 --2- 172.21.15.188:0/663857715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f00f55c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- 172.21.15.188:0/663857715 >> 172.21.15.188:0/663857715 conn(0x7f64f00f09a0 msgr2=0x7f64f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- 172.21.15.188:0/663857715 shutdown_connections 2023-11-15T04:15:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- 172.21.15.188:0/663857715 wait complete. 2023-11-15T04:15:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 Processor -- start 2023-11-15T04:15:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- start start 2023-11-15T04:15:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.411+0000 7f64ef59e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0060240 con 0x7f64f00f51a0 2023-11-15T04:15:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ee59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ee59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51662/0 (socket says 172.21.15.188:51662) 2023-11-15T04:15:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ee59c700 1 -- 172.21.15.188:0/2315374617 learned_addr learned my addr 172.21.15.188:0/2315374617 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ee59c700 1 -- 172.21.15.188:0/2315374617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e4005040 con 0x7f64f00f51a0 2023-11-15T04:15:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ee59c700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f64e4013510 tx=0x7f64e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e4009320 con 0x7f64f00f51a0 2023-11-15T04:15:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0060440 con 0x7f64f00f51a0 2023-11-15T04:15:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e4013f00 con 0x7f64f00f51a0 2023-11-15T04:15:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e4014dc0 con 0x7f64f00f51a0 2023-11-15T04:15:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.412+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f0109860 con 0x7f64f00f51a0 2023-11-15T04:15:04.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.413+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64e40095f0 con 0x7f64f00f51a0 2023-11-15T04:15:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.413+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f0045ad0 con 0x7f64f00f51a0 2023-11-15T04:15:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.413+0000 7f64deffd700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.413+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64e40410c0 con 0x7f64f00f51a0 2023-11-15T04:15:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.413+0000 7f64edd9b700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.414+0000 7f64edd9b700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 0x7f64d8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f64e00096a0 tx=0x7f64e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:04.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.418+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64e401c790 con 0x7f64f00f51a0 2023-11-15T04:15:04.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.623+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64f010a080 con 0x7f64f00f51a0 2023-11-15T04:15:04.625 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.624+0000 7f64deffd700 1 -- 172.21.15.188:0/2315374617 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64e4010230 con 0x7f64f00f51a0 2023-11-15T04:15:04.625 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:04.625 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:04.626 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 msgr2=0x7f64d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 0x7f64d8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f64e00096a0 tx=0x7f64e0006b40).stop 2023-11-15T04:15:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 msgr2=0x7f64f01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f64e4013510 tx=0x7f64e400b040).stop 2023-11-15T04:15:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 shutdown_connections 2023-11-15T04:15:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 --2- 172.21.15.188:0/2315374617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f00f51a0 0x7f64f01091b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 >> 172.21.15.188:0/2315374617 conn(0x7f64f00f09a0 msgr2=0x7f64f00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 shutdown_connections 2023-11-15T04:15:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:04.625+0000 7f64ef59e700 1 -- 172.21.15.188:0/2315374617 wait complete. 2023-11-15T04:15:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:04.103661+0000 mgr.smithi038.bnbogz (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:04.625217+0000 mon.smithi038 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.188:0/2315374617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:06.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:06.020 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:07.345 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- 172.21.15.188:0/440609206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 msgr2=0x7f57a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:07.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/440609206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a00f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5794009fb0 tx=0x7f5794009b30).stop 2023-11-15T04:15:07.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- 172.21.15.188:0/440609206 shutdown_connections 2023-11-15T04:15:07.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/440609206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a00f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:07.346 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- 172.21.15.188:0/440609206 >> 172.21.15.188:0/440609206 conn(0x7f57a00f0b70 msgr2=0x7f57a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:07.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- 172.21.15.188:0/440609206 shutdown_connections 2023-11-15T04:15:07.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- 172.21.15.188:0/440609206 wait complete. 2023-11-15T04:15:07.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 Processor -- start 2023-11-15T04:15:07.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.342+0000 7f57a6cdf700 1 -- start start 2023-11-15T04:15:07.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a6cdf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:07.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a6cdf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0106c20 con 0x7f57a00f5410 2023-11-15T04:15:07.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a4a7b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:07.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a4a7b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51692/0 (socket says 172.21.15.188:51692) 2023-11-15T04:15:07.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a4a7b700 1 -- 172.21.15.188:0/1610761400 learned_addr learned my addr 172.21.15.188:0/1610761400 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:07.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.343+0000 7f57a4a7b700 1 -- 172.21.15.188:0/1610761400 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794005040 con 0x7f57a00f5410 2023-11-15T04:15:07.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f57a4a7b700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5794013510 tx=0x7f579400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:07.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5794009320 con 0x7f57a00f5410 2023-11-15T04:15:07.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0106e20 con 0x7f57a00f5410 2023-11-15T04:15:07.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5794013f00 con 0x7f57a00f5410 2023-11-15T04:15:07.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5794014d90 con 0x7f57a00f5410 2023-11-15T04:15:07.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.344+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a01072b0 con 0x7f57a00f5410 2023-11-15T04:15:07.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57940095f0 con 0x7f57a00f5410 2023-11-15T04:15:07.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0045ad0 con 0x7f57a00f5410 2023-11-15T04:15:07.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f579d7fa700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 0x7f5790032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:07.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5794040fd0 con 0x7f57a00f5410 2023-11-15T04:15:07.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f579ffff700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 0x7f5790032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:07.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.345+0000 7f579ffff700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 0x7f5790032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:07.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.350+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5794014420 con 0x7f57a00f5410 2023-11-15T04:15:07.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.553+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57a00fae60 con 0x7f57a00f5410 2023-11-15T04:15:07.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.554+0000 7f579d7fa700 1 -- 172.21.15.188:0/1610761400 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f579401d7f0 con 0x7f57a00f5410 2023-11-15T04:15:07.555 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:07.555 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:07.556 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 msgr2=0x7f5790032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:07.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 0x7f5790032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).stop 2023-11-15T04:15:07.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 msgr2=0x7f57a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:07.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5794013510 tx=0x7f579400b040).stop 2023-11-15T04:15:07.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 shutdown_connections 2023-11-15T04:15:07.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f579002ff60 0x7f5790032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:07.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 --2- 172.21.15.188:0/1610761400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57a00f5410 0x7f57a0106730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:07.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 >> 172.21.15.188:0/1610761400 conn(0x7f57a00f0b70 msgr2=0x7f57a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:07.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 shutdown_connections 2023-11-15T04:15:07.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:07.556+0000 7f57a6cdf700 1 -- 172.21.15.188:0/1610761400 wait complete. 2023-11-15T04:15:07.559 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:06.103980+0000 mgr.smithi038.bnbogz (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:08.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:08 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:07.555550+0000 mon.smithi038 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.188:0/1610761400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:08.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:08.924 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:08.104261+0000 mgr.smithi038.bnbogz (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:10.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- 172.21.15.188:0/4154689645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 msgr2=0x7f11340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:10.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 --2- 172.21.15.188:0/4154689645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f1124009fb0 tx=0x7f1124009b30).stop 2023-11-15T04:15:10.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- 172.21.15.188:0/4154689645 shutdown_connections 2023-11-15T04:15:10.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 --2- 172.21.15.188:0/4154689645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:10.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- 172.21.15.188:0/4154689645 >> 172.21.15.188:0/4154689645 conn(0x7f11340f0b70 msgr2=0x7f11340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:10.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- 172.21.15.188:0/4154689645 shutdown_connections 2023-11-15T04:15:10.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- 172.21.15.188:0/4154689645 wait complete. 2023-11-15T04:15:10.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 Processor -- start 2023-11-15T04:15:10.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- start start 2023-11-15T04:15:10.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:10.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.283+0000 7f113c754700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340fb820 con 0x7f11340f5410 2023-11-15T04:15:10.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113a4f0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113a4f0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59656/0 (socket says 172.21.15.188:59656) 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113a4f0700 1 -- 172.21.15.188:0/3834870309 learned_addr learned my addr 172.21.15.188:0/3834870309 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113a4f0700 1 -- 172.21.15.188:0/3834870309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1124005040 con 0x7f11340f5410 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113a4f0700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1124013510 tx=0x7f112400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1124009320 con 0x7f11340f5410 2023-11-15T04:15:10.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1124013f00 con 0x7f11340f5410 2023-11-15T04:15:10.289 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11340fb9c0 con 0x7f11340f5410 2023-11-15T04:15:10.289 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1124014d90 con 0x7f11340f5410 2023-11-15T04:15:10.289 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.284+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11340fbe50 con 0x7f11340f5410 2023-11-15T04:15:10.290 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.285+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1124014420 con 0x7f11340f5410 2023-11-15T04:15:10.290 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.285+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1134059100 con 0x7f11340f5410 2023-11-15T04:15:10.290 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.285+0000 7f112affd700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 0x7f1120032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:10.291 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.285+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11240410c0 con 0x7f11340f5410 2023-11-15T04:15:10.291 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.286+0000 7f1139cef700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 0x7f1120032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:10.291 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.286+0000 7f1139cef700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 0x7f1120032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f11300096a0 tx=0x7f1130006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:10.291 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.290+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11240095f0 con 0x7f11340f5410 2023-11-15T04:15:10.500 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.499+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11340fae60 con 0x7f11340f5410 2023-11-15T04:15:10.501 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.500+0000 7f112affd700 1 -- 172.21.15.188:0/3834870309 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f112401db70 con 0x7f11340f5410 2023-11-15T04:15:10.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:10.501 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:10.502 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.501+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 msgr2=0x7f1120032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:10.502 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.501+0000 7f113c754700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 0x7f1120032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f11300096a0 tx=0x7f1130006b40).stop 2023-11-15T04:15:10.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.501+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 msgr2=0x7f11340fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:10.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.501+0000 7f113c754700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1124013510 tx=0x7f112400b040).stop 2023-11-15T04:15:10.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 shutdown_connections 2023-11-15T04:15:10.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f112002ff60 0x7f1120032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:10.503 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 --2- 172.21.15.188:0/3834870309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11340f5410 0x7f11340fd3a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:10.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 >> 172.21.15.188:0/3834870309 conn(0x7f11340f0b70 msgr2=0x7f113405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:10.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 shutdown_connections 2023-11-15T04:15:10.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:10.502+0000 7f113c754700 1 -- 172.21.15.188:0/3834870309 wait complete. 2023-11-15T04:15:10.504 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:10.104588+0000 mgr.smithi038.bnbogz (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:10.501469+0000 mon.smithi038 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.188:0/3834870309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:11.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:11.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:12.104943+0000 mgr.smithi038.bnbogz (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:13.152 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 -- 172.21.15.188:0/219005709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 msgr2=0x7fac240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:13.152 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 --2- 172.21.15.188:0/219005709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac240f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fac18009fb0 tx=0x7fac18009b30).stop 2023-11-15T04:15:13.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 -- 172.21.15.188:0/219005709 shutdown_connections 2023-11-15T04:15:13.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 --2- 172.21.15.188:0/219005709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac240f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:13.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 -- 172.21.15.188:0/219005709 >> 172.21.15.188:0/219005709 conn(0x7fac240f1c70 msgr2=0x7fac240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:13.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 -- 172.21.15.188:0/219005709 shutdown_connections 2023-11-15T04:15:13.153 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 -- 172.21.15.188:0/219005709 wait complete. 2023-11-15T04:15:13.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.150+0000 7fac2a784700 1 Processor -- start 2023-11-15T04:15:13.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac2a784700 1 -- start start 2023-11-15T04:15:13.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac2a784700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:13.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac2a784700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac24106c20 con 0x7fac240f6510 2023-11-15T04:15:13.154 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:13.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59666/0 (socket says 172.21.15.188:59666) 2023-11-15T04:15:13.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac23fff700 1 -- 172.21.15.188:0/2272553676 learned_addr learned my addr 172.21.15.188:0/2272553676 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:13.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac23fff700 1 -- 172.21.15.188:0/2272553676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18005040 con 0x7fac240f6510 2023-11-15T04:15:13.155 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac23fff700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fac18013510 tx=0x7fac1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:13.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac18009320 con 0x7fac240f6510 2023-11-15T04:15:13.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.151+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac24106e20 con 0x7fac240f6510 2023-11-15T04:15:13.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.152+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac18013f00 con 0x7fac240f6510 2023-11-15T04:15:13.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.152+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac18014d90 con 0x7fac240f6510 2023-11-15T04:15:13.156 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.152+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac241072b0 con 0x7fac240f6510 2023-11-15T04:15:13.158 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.152+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac180263a0 con 0x7fac240f6510 2023-11-15T04:15:13.158 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.152+0000 7fac20ff9700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 0x7fac0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:13.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.153+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac18040f40 con 0x7fac240f6510 2023-11-15T04:15:13.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.153+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac04004fa0 con 0x7fac240f6510 2023-11-15T04:15:13.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.153+0000 7fac237fe700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 0x7fac0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:13.159 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.153+0000 7fac237fe700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 0x7fac0c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fac140096a0 tx=0x7fac14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:13.160 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.158+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac18012920 con 0x7fac240f6510 2023-11-15T04:15:13.362 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.362+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac04005670 con 0x7fac240f6510 2023-11-15T04:15:13.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.363+0000 7fac20ff9700 1 -- 172.21.15.188:0/2272553676 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac1802b020 con 0x7fac240f6510 2023-11-15T04:15:13.363 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:13.364 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:13.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 msgr2=0x7fac0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:13.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 0x7fac0c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fac140096a0 tx=0x7fac14006b40).stop 2023-11-15T04:15:13.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 msgr2=0x7fac24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:13.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fac18013510 tx=0x7fac1800b040).stop 2023-11-15T04:15:13.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 shutdown_connections 2023-11-15T04:15:13.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fac0c02ff60 0x7fac0c032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:13.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 --2- 172.21.15.188:0/2272553676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac240f6510 0x7fac24106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:13.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 >> 172.21.15.188:0/2272553676 conn(0x7fac240f1c70 msgr2=0x7fac240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:13.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 shutdown_connections 2023-11-15T04:15:13.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:13.364+0000 7fac2a784700 1 -- 172.21.15.188:0/2272553676 wait complete. 2023-11-15T04:15:13.367 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:13.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:13.363872+0000 mon.smithi038 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.188:0/2272553676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:14.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:14.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:14.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:14.105386+0000 mgr.smithi038.bnbogz (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:16.036 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- 172.21.15.188:0/2546247087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 msgr2=0x7f759c069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:16.036 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 --2- 172.21.15.188:0/2546247087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c069ec0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f758c009fb0 tx=0x7f758c009b30).stop 2023-11-15T04:15:16.036 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- 172.21.15.188:0/2546247087 shutdown_connections 2023-11-15T04:15:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 --2- 172.21.15.188:0/2546247087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c069ec0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- 172.21.15.188:0/2546247087 >> 172.21.15.188:0/2546247087 conn(0x7f759c0f1c70 msgr2=0x7f759c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- 172.21.15.188:0/2546247087 shutdown_connections 2023-11-15T04:15:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- 172.21.15.188:0/2546247087 wait complete. 2023-11-15T04:15:16.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 Processor -- start 2023-11-15T04:15:16.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.034+0000 7f75a4466700 1 -- start start 2023-11-15T04:15:16.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a4466700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:16.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a4466700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c10d220 con 0x7f759c06b9c0 2023-11-15T04:15:16.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a2202700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:16.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a2202700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:59682/0 (socket says 172.21.15.188:59682) 2023-11-15T04:15:16.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a2202700 1 -- 172.21.15.188:0/2109527503 learned_addr learned my addr 172.21.15.188:0/2109527503 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:16.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a2202700 1 -- 172.21.15.188:0/2109527503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c005040 con 0x7f759c06b9c0 2023-11-15T04:15:16.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a2202700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f758c013510 tx=0x7f758c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:16.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758c009320 con 0x7f759c06b9c0 2023-11-15T04:15:16.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f758c013f00 con 0x7f759c06b9c0 2023-11-15T04:15:16.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758c014dc0 con 0x7f759c06b9c0 2023-11-15T04:15:16.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.035+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c10d420 con 0x7f759c06b9c0 2023-11-15T04:15:16.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.036+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c10dfd0 con 0x7f759c06b9c0 2023-11-15T04:15:16.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.036+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f758c0095f0 con 0x7f759c06b9c0 2023-11-15T04:15:16.041 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.036+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c059100 con 0x7f759c06b9c0 2023-11-15T04:15:16.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.036+0000 7f7592ffd700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:16.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.037+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f758c041000 con 0x7f759c06b9c0 2023-11-15T04:15:16.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.037+0000 7f75a1a01700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:16.042 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.037+0000 7f75a1a01700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 0x7f7588032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f75980096a0 tx=0x7f7598006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:16.043 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.042+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f758c014420 con 0x7f759c06b9c0 2023-11-15T04:15:16.244 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.244+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f759c10e660 con 0x7f759c06b9c0 2023-11-15T04:15:16.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.244+0000 7f7592ffd700 1 -- 172.21.15.188:0/2109527503 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f758c010230 con 0x7f759c06b9c0 2023-11-15T04:15:16.245 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:16.246 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:16.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 msgr2=0x7f7588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:16.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 0x7f7588032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f75980096a0 tx=0x7f7598006b40).stop 2023-11-15T04:15:16.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 msgr2=0x7f759c10cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:16.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f758c013510 tx=0x7f758c00b040).stop 2023-11-15T04:15:16.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 shutdown_connections 2023-11-15T04:15:16.248 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:16.248 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 --2- 172.21.15.188:0/2109527503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f759c06b9c0 0x7f759c10cd30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:16.248 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 >> 172.21.15.188:0/2109527503 conn(0x7f759c0f1c70 msgr2=0x7f759c0f2750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:16.248 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 shutdown_connections 2023-11-15T04:15:16.248 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:16.246+0000 7f75a4466700 1 -- 172.21.15.188:0/2109527503 wait complete. 2023-11-15T04:15:16.249 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:16.105806+0000 mgr.smithi038.bnbogz (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:16.245934+0000 mon.smithi038 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.188:0/2109527503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:17.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:17.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:18.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- 172.21.15.188:0/1082951464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 msgr2=0x7f4f180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:18.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/1082951464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f180f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4f0c009fb0 tx=0x7f4f0c009b30).stop 2023-11-15T04:15:18.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- 172.21.15.188:0/1082951464 shutdown_connections 2023-11-15T04:15:18.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/1082951464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f180f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:18.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- 172.21.15.188:0/1082951464 >> 172.21.15.188:0/1082951464 conn(0x7f4f180f0b70 msgr2=0x7f4f180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:18.893 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- 172.21.15.188:0/1082951464 shutdown_connections 2023-11-15T04:15:18.893 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- 172.21.15.188:0/1082951464 wait complete. 2023-11-15T04:15:18.893 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 Processor -- start 2023-11-15T04:15:18.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.891+0000 7f4f1caa5700 1 -- start start 2023-11-15T04:15:18.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1caa5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:18.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1caa5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18105b20 con 0x7f4f180f5410 2023-11-15T04:15:18.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:18.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49248/0 (socket says 172.21.15.188:49248) 2023-11-15T04:15:18.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1659c700 1 -- 172.21.15.188:0/2885511489 learned_addr learned my addr 172.21.15.188:0/2885511489 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:18.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1659c700 1 -- 172.21.15.188:0/2885511489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c005040 con 0x7f4f180f5410 2023-11-15T04:15:18.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.892+0000 7f4f1659c700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4f0c013510 tx=0x7f4f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:18.897 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.893+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f0c009320 con 0x7f4f180f5410 2023-11-15T04:15:18.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.893+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18105d20 con 0x7f4f180f5410 2023-11-15T04:15:18.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.893+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f0c013f00 con 0x7f4f180f5410 2023-11-15T04:15:18.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.893+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f0c014d90 con 0x7f4f180f5410 2023-11-15T04:15:18.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.893+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f181061b0 con 0x7f4f180f5410 2023-11-15T04:15:18.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f0c0095f0 con 0x7f4f180f5410 2023-11-15T04:15:18.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18045ad0 con 0x7f4f180f5410 2023-11-15T04:15:18.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f06ffd700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 0x7f4f00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:18.899 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f0c0410a0 con 0x7f4f180f5410 2023-11-15T04:15:18.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f15d9b700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 0x7f4f00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:18.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.894+0000 7f4f15d9b700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 0x7f4f00032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4f080096a0 tx=0x7f4f08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:18.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:18.899+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f0c01d7f0 con 0x7f4f180f5410 2023-11-15T04:15:19.102 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.102+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f180fae60 con 0x7f4f180f5410 2023-11-15T04:15:19.102 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.103+0000 7f4f06ffd700 1 -- 172.21.15.188:0/2885511489 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f0c010230 con 0x7f4f180f5410 2023-11-15T04:15:19.103 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:19.103 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:19.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 msgr2=0x7f4f00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:19.104 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 0x7f4f00032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4f080096a0 tx=0x7f4f08006b40).stop 2023-11-15T04:15:19.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 msgr2=0x7f4f18105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:19.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4f0c013510 tx=0x7f4f0c00b040).stop 2023-11-15T04:15:19.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 shutdown_connections 2023-11-15T04:15:19.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4f0002ff60 0x7f4f00032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:19.105 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 --2- 172.21.15.188:0/2885511489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f180f5410 0x7f4f18105630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:19.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 >> 172.21.15.188:0/2885511489 conn(0x7f4f180f0b70 msgr2=0x7f4f180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:19.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 shutdown_connections 2023-11-15T04:15:19.106 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:19.104+0000 7f4f1caa5700 1 -- 172.21.15.188:0/2885511489 wait complete. 2023-11-15T04:15:19.106 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:18.106156+0000 mgr.smithi038.bnbogz (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:19.103050+0000 mon.smithi038 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.188:0/2885511489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:20.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:20.507 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:20.106440+0000 mgr.smithi038.bnbogz (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:21.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 -- 172.21.15.188:0/2224721662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 msgr2=0x7f780c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:21.804 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 --2- 172.21.15.188:0/2224721662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c0f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7804009960 tx=0x7f7804004d10).stop 2023-11-15T04:15:21.804 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 -- 172.21.15.188:0/2224721662 shutdown_connections 2023-11-15T04:15:21.804 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 --2- 172.21.15.188:0/2224721662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c0f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:21.804 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 -- 172.21.15.188:0/2224721662 >> 172.21.15.188:0/2224721662 conn(0x7f780c0f1c70 msgr2=0x7f780c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:21.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 -- 172.21.15.188:0/2224721662 shutdown_connections 2023-11-15T04:15:21.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.803+0000 7f78117bf700 1 -- 172.21.15.188:0/2224721662 wait complete. 2023-11-15T04:15:21.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.804+0000 7f78117bf700 1 Processor -- start 2023-11-15T04:15:21.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.804+0000 7f78117bf700 1 -- start start 2023-11-15T04:15:21.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.804+0000 7f78117bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.804+0000 7f78117bf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c069e20 con 0x7f780c0f6510 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f780affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f780affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49262/0 (socket says 172.21.15.188:49262) 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f780affd700 1 -- 172.21.15.188:0/2637924895 learned_addr learned my addr 172.21.15.188:0/2637924895 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f780affd700 1 -- 172.21.15.188:0/2637924895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804009640 con 0x7f780c0f6510 2023-11-15T04:15:21.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f780affd700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7804004b70 tx=0x7f7804004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:21.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7804018070 con 0x7f780c0f6510 2023-11-15T04:15:21.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f780400b870 con 0x7f780c0f6510 2023-11-15T04:15:21.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f780401c560 con 0x7f780c0f6510 2023-11-15T04:15:21.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c069fc0 con 0x7f780c0f6510 2023-11-15T04:15:21.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.805+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c06a450 con 0x7f780c0f6510 2023-11-15T04:15:21.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.806+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7804013630 con 0x7f780c0f6510 2023-11-15T04:15:21.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.806+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c059100 con 0x7f780c0f6510 2023-11-15T04:15:21.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.806+0000 7f78037fe700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 0x7f77f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:21.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.806+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78040411b0 con 0x7f780c0f6510 2023-11-15T04:15:21.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.807+0000 7f780a7fc700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 0x7f77f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:21.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.807+0000 7f780a7fc700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 0x7f77f8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f77f40096a0 tx=0x7f77f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:21.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:21.811+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7804013a30 con 0x7f780c0f6510 2023-11-15T04:15:22.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.015+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f780c0fbf60 con 0x7f780c0f6510 2023-11-15T04:15:22.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.016+0000 7f78037fe700 1 -- 172.21.15.188:0/2637924895 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7804011b10 con 0x7f780c0f6510 2023-11-15T04:15:22.016 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:22.016 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:22.017 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.017+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 msgr2=0x7f77f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:22.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.017+0000 7f78117bf700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 0x7f77f8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f77f40096a0 tx=0x7f77f4006b40).stop 2023-11-15T04:15:22.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.017+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 msgr2=0x7f780c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:22.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.017+0000 7f78117bf700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7804004b70 tx=0x7f7804004d10).stop 2023-11-15T04:15:22.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 shutdown_connections 2023-11-15T04:15:22.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f77f802ffb0 0x7f77f8032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:22.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 --2- 172.21.15.188:0/2637924895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f780c0f6510 0x7f780c06b9a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:22.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 >> 172.21.15.188:0/2637924895 conn(0x7f780c0f1c70 msgr2=0x7f780c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:22.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 shutdown_connections 2023-11-15T04:15:22.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:22.018+0000 7f78117bf700 1 -- 172.21.15.188:0/2637924895 wait complete. 2023-11-15T04:15:22.020 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:22.016294+0000 mon.smithi038 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.188:0/2637924895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:22.106809+0000 mgr.smithi038.bnbogz (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:23.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:23.412 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:24.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 -- 172.21.15.188:0/779001353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 msgr2=0x7ff9e80f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:24.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/779001353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e80f5df0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff9dc009fb0 tx=0x7ff9dc009b30).stop 2023-11-15T04:15:24.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 -- 172.21.15.188:0/779001353 shutdown_connections 2023-11-15T04:15:24.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/779001353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e80f5df0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:24.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 -- 172.21.15.188:0/779001353 >> 172.21.15.188:0/779001353 conn(0x7ff9e80f02e0 msgr2=0x7ff9e80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:24.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 -- 172.21.15.188:0/779001353 shutdown_connections 2023-11-15T04:15:24.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.679+0000 7ff9ed81b700 1 -- 172.21.15.188:0/779001353 wait complete. 2023-11-15T04:15:24.682 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.680+0000 7ff9ed81b700 1 Processor -- start 2023-11-15T04:15:24.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.680+0000 7ff9ed81b700 1 -- start start 2023-11-15T04:15:24.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.680+0000 7ff9ed81b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:24.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.680+0000 7ff9ed81b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8105ab0 con 0x7ff9e80f59d0 2023-11-15T04:15:24.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:24.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49278/0 (socket says 172.21.15.188:49278) 2023-11-15T04:15:24.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9e6ffd700 1 -- 172.21.15.188:0/3796490373 learned_addr learned my addr 172.21.15.188:0/3796490373 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:24.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9e6ffd700 1 -- 172.21.15.188:0/3796490373 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc005040 con 0x7ff9e80f59d0 2023-11-15T04:15:24.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9e6ffd700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff9dc013510 tx=0x7ff9dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:24.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc009320 con 0x7ff9e80f59d0 2023-11-15T04:15:24.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9dc013f00 con 0x7ff9e80f59d0 2023-11-15T04:15:24.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8105cb0 con 0x7ff9e80f59d0 2023-11-15T04:15:24.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.681+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc014d90 con 0x7ff9e80f59d0 2023-11-15T04:15:24.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.682+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e8106140 con 0x7ff9e80f59d0 2023-11-15T04:15:24.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.682+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9dc0263a0 con 0x7ff9e80f59d0 2023-11-15T04:15:24.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.682+0000 7ff9cf7fe700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 0x7ff9d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:24.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.682+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9dc041010 con 0x7ff9e80f59d0 2023-11-15T04:15:24.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.682+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8045ad0 con 0x7ff9e80f59d0 2023-11-15T04:15:24.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.683+0000 7ff9e67fc700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 0x7ff9d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:24.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.683+0000 7ff9e67fc700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 0x7ff9d0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff9d80096a0 tx=0x7ff9d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:24.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.687+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9dc012920 con 0x7ff9e80f59d0 2023-11-15T04:15:24.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.892+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9e80faf40 con 0x7ff9e80f59d0 2023-11-15T04:15:24.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.893+0000 7ff9cf7fe700 1 -- 172.21.15.188:0/3796490373 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9dc02b020 con 0x7ff9e80f59d0 2023-11-15T04:15:24.893 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:24.893 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:24.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 msgr2=0x7ff9d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:24.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 0x7ff9d0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff9d80096a0 tx=0x7ff9d8006b40).stop 2023-11-15T04:15:24.894 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 msgr2=0x7ff9e81055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:24.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff9dc013510 tx=0x7ff9dc00b040).stop 2023-11-15T04:15:24.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 shutdown_connections 2023-11-15T04:15:24.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff9d002ff60 0x7ff9d0032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:24.895 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 --2- 172.21.15.188:0/3796490373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e80f59d0 0x7ff9e81055c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:24.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 >> 172.21.15.188:0/3796490373 conn(0x7ff9e80f02e0 msgr2=0x7ff9e80f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:24.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 shutdown_connections 2023-11-15T04:15:24.896 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:24.894+0000 7ff9ed81b700 1 -- 172.21.15.188:0/3796490373 wait complete. 2023-11-15T04:15:24.896 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:24.107218+0000 mgr.smithi038.bnbogz (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:24.892984+0000 mon.smithi038 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.188:0/3796490373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:26.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:26.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:27.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- 172.21.15.188:0/3848508635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 msgr2=0x7fe25c0f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:27.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 --2- 172.21.15.188:0/3848508635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c0f5d70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe24c009fb0 tx=0x7fe24c009b30).stop 2023-11-15T04:15:27.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- 172.21.15.188:0/3848508635 shutdown_connections 2023-11-15T04:15:27.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 --2- 172.21.15.188:0/3848508635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c0f5d70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:27.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- 172.21.15.188:0/3848508635 >> 172.21.15.188:0/3848508635 conn(0x7fe25c0f0260 msgr2=0x7fe25c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:27.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- 172.21.15.188:0/3848508635 shutdown_connections 2023-11-15T04:15:27.522 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- 172.21.15.188:0/3848508635 wait complete. 2023-11-15T04:15:27.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 Processor -- start 2023-11-15T04:15:27.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.520+0000 7fe263c29700 1 -- start start 2023-11-15T04:15:27.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe263c29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:27.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe263c29700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c106c30 con 0x7fe25c0f5950 2023-11-15T04:15:27.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe2619c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe2619c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:49294/0 (socket says 172.21.15.188:49294) 2023-11-15T04:15:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe2619c5700 1 -- 172.21.15.188:0/1734917648 learned_addr learned my addr 172.21.15.188:0/1734917648 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe2619c5700 1 -- 172.21.15.188:0/1734917648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c005040 con 0x7fe25c0f5950 2023-11-15T04:15:27.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.521+0000 7fe2619c5700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe24c014000 tx=0x7fe24c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:27.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.522+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe24c0095f0 con 0x7fe25c0f5950 2023-11-15T04:15:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.522+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c106e30 con 0x7fe25c0f5950 2023-11-15T04:15:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.522+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe24c007e70 con 0x7fe25c0f5950 2023-11-15T04:15:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.522+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe24c014ea0 con 0x7fe25c0f5950 2023-11-15T04:15:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.522+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c1072c0 con 0x7fe25c0f5950 2023-11-15T04:15:27.526 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c045ad0 con 0x7fe25c0f5950 2023-11-15T04:15:27.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe24c014690 con 0x7fe25c0f5950 2023-11-15T04:15:27.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe2527fc700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 0x7fe248036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:27.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe24c040870 con 0x7fe25c0f5950 2023-11-15T04:15:27.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe2611c4700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 0x7fe248036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:27.527 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.523+0000 7fe2611c4700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 0x7fe248036830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe2580096a0 tx=0x7fe258006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:27.528 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.528+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe24c00f300 con 0x7fe25c0f5950 2023-11-15T04:15:27.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:26.107602+0000 mgr.smithi038.bnbogz (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:27.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.732+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe25c108170 con 0x7fe25c0f5950 2023-11-15T04:15:27.732 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.732+0000 7fe2527fc700 1 -- 172.21.15.188:0/1734917648 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe24c00f300 con 0x7fe25c0f5950 2023-11-15T04:15:27.732 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:27.733 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:27.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 msgr2=0x7fe248036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:27.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 0x7fe248036830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe2580096a0 tx=0x7fe258006b40).stop 2023-11-15T04:15:27.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 msgr2=0x7fe25c106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:27.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe24c014000 tx=0x7fe24c00b040).stop 2023-11-15T04:15:27.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 shutdown_connections 2023-11-15T04:15:27.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe248034370 0x7fe248036830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:27.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 --2- 172.21.15.188:0/1734917648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe25c0f5950 0x7fe25c106740 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:27.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 >> 172.21.15.188:0/1734917648 conn(0x7fe25c0f0260 msgr2=0x7fe25c0f4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:27.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 shutdown_connections 2023-11-15T04:15:27.736 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:27.734+0000 7fe263c29700 1 -- 172.21.15.188:0/1734917648 wait complete. 2023-11-15T04:15:27.736 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:28.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:27.732751+0000 mon.smithi038 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.188:0/1734917648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:29.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:29.108 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:28.107924+0000 mgr.smithi038.bnbogz (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:30.299 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- 172.21.15.188:0/4130900108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 msgr2=0x7f6c780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:30.299 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 --2- 172.21.15.188:0/4130900108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c780f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6c68009fb0 tx=0x7f6c68009b30).stop 2023-11-15T04:15:30.300 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- 172.21.15.188:0/4130900108 shutdown_connections 2023-11-15T04:15:30.300 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 --2- 172.21.15.188:0/4130900108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c780f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:30.300 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- 172.21.15.188:0/4130900108 >> 172.21.15.188:0/4130900108 conn(0x7f6c780f09a0 msgr2=0x7f6c780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:30.300 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- 172.21.15.188:0/4130900108 shutdown_connections 2023-11-15T04:15:30.300 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- 172.21.15.188:0/4130900108 wait complete. 2023-11-15T04:15:30.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 Processor -- start 2023-11-15T04:15:30.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.298+0000 7f6c7d098700 1 -- start start 2023-11-15T04:15:30.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c7d098700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:30.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c7d098700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78060220 con 0x7f6c780f51a0 2023-11-15T04:15:30.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c77fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:30.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c77fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46926/0 (socket says 172.21.15.188:46926) 2023-11-15T04:15:30.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c77fff700 1 -- 172.21.15.188:0/1656999029 learned_addr learned my addr 172.21.15.188:0/1656999029 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:30.304 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c77fff700 1 -- 172.21.15.188:0/1656999029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c68005040 con 0x7f6c780f51a0 2023-11-15T04:15:30.304 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.299+0000 7f6c77fff700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f6c68013510 tx=0x7f6c6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:30.304 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c68009320 con 0x7f6c780f51a0 2023-11-15T04:15:30.304 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c78060420 con 0x7f6c780f51a0 2023-11-15T04:15:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c68013f00 con 0x7f6c780f51a0 2023-11-15T04:15:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c68014dc0 con 0x7f6c780f51a0 2023-11-15T04:15:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c78109880 con 0x7f6c780f51a0 2023-11-15T04:15:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c680095f0 con 0x7f6c780f51a0 2023-11-15T04:15:30.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.300+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c78059100 con 0x7f6c780f51a0 2023-11-15T04:15:30.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.301+0000 7f6c74ff9700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 0x7f6c60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:30.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.301+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c68041040 con 0x7f6c780f51a0 2023-11-15T04:15:30.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.301+0000 7f6c777fe700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 0x7f6c60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:30.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.301+0000 7f6c777fe700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 0x7f6c60032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f6c6c0096a0 tx=0x7f6c6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:30.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.305+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c68014420 con 0x7f6c780f51a0 2023-11-15T04:15:30.512 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.512+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c78045ad0 con 0x7f6c780f51a0 2023-11-15T04:15:30.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.513+0000 7f6c74ff9700 1 -- 172.21.15.188:0/1656999029 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c68010230 con 0x7f6c780f51a0 2023-11-15T04:15:30.513 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:30.514 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:30.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 msgr2=0x7f6c60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:30.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 0x7f6c60032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f6c6c0096a0 tx=0x7f6c6c006b40).stop 2023-11-15T04:15:30.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 msgr2=0x7f6c781091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f6c68013510 tx=0x7f6c6800b040).stop 2023-11-15T04:15:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 shutdown_connections 2023-11-15T04:15:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6c6002ff60 0x7f6c60032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 --2- 172.21.15.188:0/1656999029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c780f51a0 0x7f6c781091d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:30.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 >> 172.21.15.188:0/1656999029 conn(0x7f6c780f09a0 msgr2=0x7f6c780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:30.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 shutdown_connections 2023-11-15T04:15:30.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:30.514+0000 7f6c7d098700 1 -- 172.21.15.188:0/1656999029 wait complete. 2023-11-15T04:15:30.517 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:30.108288+0000 mgr.smithi038.bnbogz (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:31.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:30.513775+0000 mon.smithi038 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.188:0/1656999029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:31.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:31.920 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:32.108640+0000 mgr.smithi038.bnbogz (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:33.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 -- 172.21.15.188:0/2552034449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 msgr2=0x7f8c700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:33.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 --2- 172.21.15.188:0/2552034449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c700f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8c64009fb0 tx=0x7f8c64009b30).stop 2023-11-15T04:15:33.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 -- 172.21.15.188:0/2552034449 shutdown_connections 2023-11-15T04:15:33.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 --2- 172.21.15.188:0/2552034449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c700f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:33.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 -- 172.21.15.188:0/2552034449 >> 172.21.15.188:0/2552034449 conn(0x7f8c700f0b70 msgr2=0x7f8c700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:33.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 -- 172.21.15.188:0/2552034449 shutdown_connections 2023-11-15T04:15:33.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.252+0000 7f8c76633700 1 -- 172.21.15.188:0/2552034449 wait complete. 2023-11-15T04:15:33.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.253+0000 7f8c76633700 1 Processor -- start 2023-11-15T04:15:33.256 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.253+0000 7f8c76633700 1 -- start start 2023-11-15T04:15:33.257 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.253+0000 7f8c76633700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:33.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.253+0000 7f8c76633700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70060240 con 0x7f8c700f5410 2023-11-15T04:15:33.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:33.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46944/0 (socket says 172.21.15.188:46944) 2023-11-15T04:15:33.258 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6ffff700 1 -- 172.21.15.188:0/1718484790 learned_addr learned my addr 172.21.15.188:0/1718484790 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:33.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6ffff700 1 -- 172.21.15.188:0/1718484790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c64005040 con 0x7f8c700f5410 2023-11-15T04:15:33.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6ffff700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8c64014000 tx=0x7f8c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:33.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c640095f0 con 0x7f8c700f5410 2023-11-15T04:15:33.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c70060440 con 0x7f8c700f5410 2023-11-15T04:15:33.259 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c70109ae0 con 0x7f8c700f5410 2023-11-15T04:15:33.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c64007e70 con 0x7f8c700f5410 2023-11-15T04:15:33.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.254+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c64014ea0 con 0x7f8c700f5410 2023-11-15T04:15:33.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.255+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c64025070 con 0x7f8c700f5410 2023-11-15T04:15:33.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.255+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c70045ad0 con 0x7f8c700f5410 2023-11-15T04:15:33.260 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.255+0000 7f8c6cff9700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:33.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.255+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c64040a30 con 0x7f8c700f5410 2023-11-15T04:15:33.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.256+0000 7f8c6f7fe700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:33.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.256+0000 7f8c6f7fe700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 0x7f8c58032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8c600096a0 tx=0x7f8c60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:33.261 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.260+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c6400f4e0 con 0x7f8c700f5410 2023-11-15T04:15:33.464 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.464+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c700fae60 con 0x7f8c700f5410 2023-11-15T04:15:33.465 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.464+0000 7f8c6cff9700 1 -- 172.21.15.188:0/1718484790 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c6400f300 con 0x7f8c700f5410 2023-11-15T04:15:33.465 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:33.466 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:33.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 msgr2=0x7f8c58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:33.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 0x7f8c58032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8c600096a0 tx=0x7f8c60006b40).stop 2023-11-15T04:15:33.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 msgr2=0x7f8c70109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:33.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8c64014000 tx=0x7f8c6400b040).stop 2023-11-15T04:15:33.467 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 shutdown_connections 2023-11-15T04:15:33.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:33.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 --2- 172.21.15.188:0/1718484790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c700f5410 0x7f8c70109430 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:33.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 >> 172.21.15.188:0/1718484790 conn(0x7f8c700f0b70 msgr2=0x7f8c700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:33.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 shutdown_connections 2023-11-15T04:15:33.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:33.466+0000 7f8c76633700 1 -- 172.21.15.188:0/1718484790 wait complete. 2023-11-15T04:15:33.469 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:33.465767+0000 mon.smithi038 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.188:0/1718484790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:34.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:34.109045+0000 mgr.smithi038.bnbogz (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:34.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:34.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:36.044 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- 172.21.15.188:0/976196717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 msgr2=0x7fd2fc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:36.044 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 --2- 172.21.15.188:0/976196717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f4be0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd2f0009fb0 tx=0x7fd2f0009b30).stop 2023-11-15T04:15:36.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- 172.21.15.188:0/976196717 shutdown_connections 2023-11-15T04:15:36.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 --2- 172.21.15.188:0/976196717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f4be0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:36.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- 172.21.15.188:0/976196717 >> 172.21.15.188:0/976196717 conn(0x7fd2fc0f02e0 msgr2=0x7fd2fc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:36.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- 172.21.15.188:0/976196717 shutdown_connections 2023-11-15T04:15:36.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- 172.21.15.188:0/976196717 wait complete. 2023-11-15T04:15:36.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 Processor -- start 2023-11-15T04:15:36.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 -- start start 2023-11-15T04:15:36.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.042+0000 7fd301dd1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:36.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd301dd1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0f57f0 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2fb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:36.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2fb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46956/0 (socket says 172.21.15.188:46956) 2023-11-15T04:15:36.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2fb7fe700 1 -- 172.21.15.188:0/2879532350 learned_addr learned my addr 172.21.15.188:0/2879532350 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:36.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2fb7fe700 1 -- 172.21.15.188:0/2879532350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0005040 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2fb7fe700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd2f0013510 tx=0x7fd2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:36.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f0009320 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc0f5990 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2f0013f00 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2f0014d90 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.043+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc0f5e20 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.044+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2f00263a0 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.044+0000 7fd2e3fff700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 0x7fd2e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:36.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.044+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2f0041080 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.044+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc059100 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.045+0000 7fd2faffd700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 0x7fd2e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:36.051 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.045+0000 7fd2faffd700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 0x7fd2e4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd2ec0096a0 tx=0x7fd2ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:36.052 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.049+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2f0012920 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.251+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2fc045ad0 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.252 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.251+0000 7fd2e3fff700 1 -- 172.21.15.188:0/2879532350 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd2f002b020 con 0x7fd2fc0f47c0 2023-11-15T04:15:36.253 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:36.253 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:36.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 msgr2=0x7fd2e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:36.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 0x7fd2e4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd2ec0096a0 tx=0x7fd2ec006b40).stop 2023-11-15T04:15:36.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 msgr2=0x7fd2fc0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:36.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd2f0013510 tx=0x7fd2f000b040).stop 2023-11-15T04:15:36.254 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 shutdown_connections 2023-11-15T04:15:36.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd2e402ff60 0x7fd2e4032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:36.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 --2- 172.21.15.188:0/2879532350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc0f47c0 0x7fd2fc0f7370 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:36.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 >> 172.21.15.188:0/2879532350 conn(0x7fd2fc0f02e0 msgr2=0x7fd2fc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:36.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 shutdown_connections 2023-11-15T04:15:36.255 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:36.253+0000 7fd301dd1700 1 -- 172.21.15.188:0/2879532350 wait complete. 2023-11-15T04:15:36.256 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:36.109382+0000 mgr.smithi038.bnbogz (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:36.253000+0000 mon.smithi038 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.188:0/2879532350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:37.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:37.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:38.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.793+0000 7f06ba66b700 1 -- 172.21.15.188:0/524753747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 msgr2=0x7f06b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:38.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.793+0000 7f06ba66b700 1 --2- 172.21.15.188:0/524753747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b40f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f06a4009fb0 tx=0x7f06a4009b30).stop 2023-11-15T04:15:38.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.793+0000 7f06ba66b700 1 -- 172.21.15.188:0/524753747 shutdown_connections 2023-11-15T04:15:38.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.793+0000 7f06ba66b700 1 --2- 172.21.15.188:0/524753747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b40f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:38.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.793+0000 7f06ba66b700 1 -- 172.21.15.188:0/524753747 >> 172.21.15.188:0/524753747 conn(0x7f06b40f0b70 msgr2=0x7f06b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:38.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 -- 172.21.15.188:0/524753747 shutdown_connections 2023-11-15T04:15:38.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 -- 172.21.15.188:0/524753747 wait complete. 2023-11-15T04:15:38.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 Processor -- start 2023-11-15T04:15:38.796 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 -- start start 2023-11-15T04:15:38.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:38.798 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.794+0000 7f06ba66b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b4105b20 con 0x7f06b40f5410 2023-11-15T04:15:38.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:38.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:52476/0 (socket says 172.21.15.188:52476) 2023-11-15T04:15:38.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b3fff700 1 -- 172.21.15.188:0/2737049634 learned_addr learned my addr 172.21.15.188:0/2737049634 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:38.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b3fff700 1 -- 172.21.15.188:0/2737049634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4005040 con 0x7f06b40f5410 2023-11-15T04:15:38.799 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b3fff700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f06a4014000 tx=0x7f06a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:38.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.795+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06a40095f0 con 0x7f06b40f5410 2023-11-15T04:15:38.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b4105d20 con 0x7f06b40f5410 2023-11-15T04:15:38.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06a4007e70 con 0x7f06b40f5410 2023-11-15T04:15:38.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06a4014ea0 con 0x7f06b40f5410 2023-11-15T04:15:38.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b41061b0 con 0x7f06b40f5410 2023-11-15T04:15:38.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06a4025070 con 0x7f06b40f5410 2023-11-15T04:15:38.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b4045ad0 con 0x7f06b40f5410 2023-11-15T04:15:38.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.796+0000 7f06b0ff9700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 0x7f069c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:38.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.797+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06a4040a00 con 0x7f06b40f5410 2023-11-15T04:15:38.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.797+0000 7f06b37fe700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 0x7f069c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:38.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.797+0000 7f06b37fe700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 0x7f069c032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f06a80096a0 tx=0x7f06a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:38.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:38.801+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06a40079c0 con 0x7f06b40f5410 2023-11-15T04:15:39.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.007+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06b40fae60 con 0x7f06b40f5410 2023-11-15T04:15:39.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.007+0000 7f06b0ff9700 1 -- 172.21.15.188:0/2737049634 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06a400f300 con 0x7f06b40f5410 2023-11-15T04:15:39.008 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:39.009 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:39.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 msgr2=0x7f069c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:39.010 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 0x7f069c032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f06a80096a0 tx=0x7f06a8006b40).stop 2023-11-15T04:15:39.010 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 msgr2=0x7f06b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:39.010 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f06a4014000 tx=0x7f06a400b040).stop 2023-11-15T04:15:39.010 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 shutdown_connections 2023-11-15T04:15:39.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f069c02ff60 0x7f069c032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:39.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 --2- 172.21.15.188:0/2737049634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06b40f5410 0x7f06b4105630 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:39.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 >> 172.21.15.188:0/2737049634 conn(0x7f06b40f0b70 msgr2=0x7f06b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:39.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 shutdown_connections 2023-11-15T04:15:39.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:39.009+0000 7f06ba66b700 1 -- 172.21.15.188:0/2737049634 wait complete. 2023-11-15T04:15:39.012 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:38.109733+0000 mgr.smithi038.bnbogz (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:39.008582+0000 mon.smithi038 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.188:0/2737049634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:40.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:40.409 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:40.110068+0000 mgr.smithi038.bnbogz (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:41.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2320067018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 msgr2=0x7f1e5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:41.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2320067018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c0f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f1e50009fb0 tx=0x7f1e50009b30).stop 2023-11-15T04:15:41.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2320067018 shutdown_connections 2023-11-15T04:15:41.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2320067018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c0f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:41.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2320067018 >> 172.21.15.188:0/2320067018 conn(0x7f1e5c0f0b70 msgr2=0x7f1e5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:41.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2320067018 shutdown_connections 2023-11-15T04:15:41.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.803+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2320067018 wait complete. 2023-11-15T04:15:41.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e60a8a700 1 Processor -- start 2023-11-15T04:15:41.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e60a8a700 1 -- start start 2023-11-15T04:15:41.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e60a8a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:41.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e60a8a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c106c00 con 0x7f1e5c0f5410 2023-11-15T04:15:41.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e5a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:41.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e5a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:52500/0 (socket says 172.21.15.188:52500) 2023-11-15T04:15:41.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e5a59c700 1 -- 172.21.15.188:0/2300997469 learned_addr learned my addr 172.21.15.188:0/2300997469 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:41.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.804+0000 7f1e5a59c700 1 -- 172.21.15.188:0/2300997469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e50005040 con 0x7f1e5c0f5410 2023-11-15T04:15:41.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e5a59c700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1e50013510 tx=0x7f1e5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:41.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e50009320 con 0x7f1e5c0f5410 2023-11-15T04:15:41.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c106e00 con 0x7f1e5c0f5410 2023-11-15T04:15:41.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e50013f00 con 0x7f1e5c0f5410 2023-11-15T04:15:41.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e50014d90 con 0x7f1e5c0f5410 2023-11-15T04:15:41.809 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c107290 con 0x7f1e5c0f5410 2023-11-15T04:15:41.810 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.805+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c045ad0 con 0x7f1e5c0f5410 2023-11-15T04:15:41.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.806+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e500095f0 con 0x7f1e5c0f5410 2023-11-15T04:15:41.811 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.806+0000 7f1e4affd700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 0x7f1e44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:41.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.806+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e50040fe0 con 0x7f1e5c0f5410 2023-11-15T04:15:41.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.806+0000 7f1e59d9b700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 0x7f1e44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:41.812 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.806+0000 7f1e59d9b700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 0x7f1e44032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f1e4c0096a0 tx=0x7f1e4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:41.813 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:41.811+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e5001d7f0 con 0x7f1e5c0f5410 2023-11-15T04:15:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.015+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e5c0fae60 con 0x7f1e5c0f5410 2023-11-15T04:15:42.016 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.016+0000 7f1e4affd700 1 -- 172.21.15.188:0/2300997469 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e50010230 con 0x7f1e5c0f5410 2023-11-15T04:15:42.017 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:42.017 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 msgr2=0x7f1e44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 0x7f1e44032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f1e4c0096a0 tx=0x7f1e4c006b40).stop 2023-11-15T04:15:42.018 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 msgr2=0x7f1e5c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:42.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1e50013510 tx=0x7f1e5000b040).stop 2023-11-15T04:15:42.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 shutdown_connections 2023-11-15T04:15:42.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e4402ff60 0x7f1e44032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:42.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 --2- 172.21.15.188:0/2300997469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e5c0f5410 0x7f1e5c106710 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:42.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 >> 172.21.15.188:0/2300997469 conn(0x7f1e5c0f0b70 msgr2=0x7f1e5c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:42.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 shutdown_connections 2023-11-15T04:15:42.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:42.017+0000 7f1e60a8a700 1 -- 172.21.15.188:0/2300997469 wait complete. 2023-11-15T04:15:42.020 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:42.017257+0000 mon.smithi038 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.188:0/2300997469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:42.110412+0000 mgr.smithi038.bnbogz (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:43.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:43.411 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:44.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.703+0000 7f9abd554700 1 -- 172.21.15.188:0/4048908170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 msgr2=0x7f9ab80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:44.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.703+0000 7f9abd554700 1 --2- 172.21.15.188:0/4048908170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab80f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f9aa8009fb0 tx=0x7f9aa8009b30).stop 2023-11-15T04:15:44.705 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.703+0000 7f9abd554700 1 -- 172.21.15.188:0/4048908170 shutdown_connections 2023-11-15T04:15:44.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.703+0000 7f9abd554700 1 --2- 172.21.15.188:0/4048908170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab80f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:44.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.703+0000 7f9abd554700 1 -- 172.21.15.188:0/4048908170 >> 172.21.15.188:0/4048908170 conn(0x7f9ab80f09a0 msgr2=0x7f9ab80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:44.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.704+0000 7f9abd554700 1 -- 172.21.15.188:0/4048908170 shutdown_connections 2023-11-15T04:15:44.706 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.704+0000 7f9abd554700 1 -- 172.21.15.188:0/4048908170 wait complete. 2023-11-15T04:15:44.707 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.704+0000 7f9abd554700 1 Processor -- start 2023-11-15T04:15:44.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.704+0000 7f9abd554700 1 -- start start 2023-11-15T04:15:44.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9abd554700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:44.709 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9abd554700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab81058c0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:44.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:52516/0 (socket says 172.21.15.188:52516) 2023-11-15T04:15:44.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab7fff700 1 -- 172.21.15.188:0/2140767480 learned_addr learned my addr 172.21.15.188:0/2140767480 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:44.710 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab7fff700 1 -- 172.21.15.188:0/2140767480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8005040 con 0x7f9ab80f51a0 2023-11-15T04:15:44.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab7fff700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f9aa8013510 tx=0x7f9aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:44.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.705+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa8009320 con 0x7f9ab80f51a0 2023-11-15T04:15:44.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab8105ac0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aa8013f00 con 0x7f9ab80f51a0 2023-11-15T04:15:44.711 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa8014dc0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab8105f50 con 0x7f9ab80f51a0 2023-11-15T04:15:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9aa8014420 con 0x7f9ab80f51a0 2023-11-15T04:15:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8045ad0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.706+0000 7f9ab4ff9700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 0x7f9aa0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:44.712 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.707+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9aa80410e0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.707+0000 7f9ab77fe700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 0x7f9aa0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:44.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.707+0000 7f9ab77fe700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 0x7f9aa0032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9aac0096a0 tx=0x7f9aac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:44.713 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.711+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9aa80095f0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.916+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ab8106da0 con 0x7f9ab80f51a0 2023-11-15T04:15:44.917 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.917+0000 7f9ab4ff9700 1 -- 172.21.15.188:0/2140767480 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9aa801cb10 con 0x7f9ab80f51a0 2023-11-15T04:15:44.918 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:44.918 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:44.919 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 msgr2=0x7f9aa0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:44.919 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 0x7f9aa0032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9aac0096a0 tx=0x7f9aac006b40).stop 2023-11-15T04:15:44.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 msgr2=0x7f9ab81053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:44.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f9aa8013510 tx=0x7f9aa800b040).stop 2023-11-15T04:15:44.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 shutdown_connections 2023-11-15T04:15:44.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9aa002ffb0 0x7f9aa0032470 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:44.920 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 --2- 172.21.15.188:0/2140767480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab80f51a0 0x7f9ab81053d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:44.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 >> 172.21.15.188:0/2140767480 conn(0x7f9ab80f09a0 msgr2=0x7f9ab80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:44.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.918+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 shutdown_connections 2023-11-15T04:15:44.921 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:44.919+0000 7f9abd554700 1 -- 172.21.15.188:0/2140767480 wait complete. 2023-11-15T04:15:44.922 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:44.110825+0000 mgr.smithi038.bnbogz (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:44.918182+0000 mon.smithi038 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.188:0/2140767480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:46.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:46.315 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:47.551 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 -- 172.21.15.188:0/971384130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 msgr2=0x7f4c980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:47.551 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/971384130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c980f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f4c8c009fb0 tx=0x7f4c8c009b30).stop 2023-11-15T04:15:47.551 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 -- 172.21.15.188:0/971384130 shutdown_connections 2023-11-15T04:15:47.551 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/971384130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c980f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:47.552 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 -- 172.21.15.188:0/971384130 >> 172.21.15.188:0/971384130 conn(0x7f4c980f0b70 msgr2=0x7f4c980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:47.552 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 -- 172.21.15.188:0/971384130 shutdown_connections 2023-11-15T04:15:47.552 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.549+0000 7f4c9edc0700 1 -- 172.21.15.188:0/971384130 wait complete. 2023-11-15T04:15:47.552 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.550+0000 7f4c9edc0700 1 Processor -- start 2023-11-15T04:15:47.552 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.550+0000 7f4c9edc0700 1 -- start start 2023-11-15T04:15:47.554 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9edc0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:47.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9edc0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98105b20 con 0x7f4c980f5410 2023-11-15T04:15:47.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9cb5c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:47.555 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9cb5c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:52538/0 (socket says 172.21.15.188:52538) 2023-11-15T04:15:47.556 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9cb5c700 1 -- 172.21.15.188:0/1594049772 learned_addr learned my addr 172.21.15.188:0/1594049772 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:47.556 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.551+0000 7f4c9cb5c700 1 -- 172.21.15.188:0/1594049772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c005040 con 0x7f4c980f5410 2023-11-15T04:15:47.556 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c9cb5c700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4c8c013510 tx=0x7f4c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:47.556 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c8c009320 con 0x7f4c980f5410 2023-11-15T04:15:47.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c8c013f00 con 0x7f4c980f5410 2023-11-15T04:15:47.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c98105d20 con 0x7f4c980f5410 2023-11-15T04:15:47.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c8c014d90 con 0x7f4c980f5410 2023-11-15T04:15:47.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.552+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981061b0 con 0x7f4c980f5410 2023-11-15T04:15:47.557 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.553+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c8c0095f0 con 0x7f4c980f5410 2023-11-15T04:15:47.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.553+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c98045ad0 con 0x7f4c980f5410 2023-11-15T04:15:47.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.553+0000 7f4c957fa700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 0x7f4c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:47.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.553+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c8c0410a0 con 0x7f4c980f5410 2023-11-15T04:15:47.558 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.553+0000 7f4c97fff700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 0x7f4c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:47.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.554+0000 7f4c97fff700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 0x7f4c88032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4c840096a0 tx=0x7f4c84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:47.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.558+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c8c01d7f0 con 0x7f4c980f5410 2023-11-15T04:15:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:46.111210+0000 mgr.smithi038.bnbogz (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:47.761 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.761+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c980fae60 con 0x7f4c980f5410 2023-11-15T04:15:47.762 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.761+0000 7f4c957fa700 1 -- 172.21.15.188:0/1594049772 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c8c010230 con 0x7f4c980f5410 2023-11-15T04:15:47.762 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:47.763 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:47.764 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 msgr2=0x7f4c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:47.764 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 0x7f4c88032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4c840096a0 tx=0x7f4c84006b40).stop 2023-11-15T04:15:47.764 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 msgr2=0x7f4c98105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:47.764 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4c8c013510 tx=0x7f4c8c00b040).stop 2023-11-15T04:15:47.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 shutdown_connections 2023-11-15T04:15:47.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4c8802ff60 0x7f4c88032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:47.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 --2- 172.21.15.188:0/1594049772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c980f5410 0x7f4c98105630 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:47.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 >> 172.21.15.188:0/1594049772 conn(0x7f4c980f0b70 msgr2=0x7f4c980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:47.765 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 shutdown_connections 2023-11-15T04:15:47.766 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:47.763+0000 7f4c9edc0700 1 -- 172.21.15.188:0/1594049772 wait complete. 2023-11-15T04:15:47.766 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:48.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:48 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:47.762701+0000 mon.smithi038 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.188:0/1594049772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:49.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:49.153 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:48.111557+0000 mgr.smithi038.bnbogz (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:50.402 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 -- 172.21.15.188:0/2962101948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 msgr2=0x7ff8540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 --2- 172.21.15.188:0/2962101948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff8540f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff844009fb0 tx=0x7ff844009b30).stop 2023-11-15T04:15:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 -- 172.21.15.188:0/2962101948 shutdown_connections 2023-11-15T04:15:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 --2- 172.21.15.188:0/2962101948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff8540f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:50.403 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 -- 172.21.15.188:0/2962101948 >> 172.21.15.188:0/2962101948 conn(0x7ff8540f1c70 msgr2=0x7ff8540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:50.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 -- 172.21.15.188:0/2962101948 shutdown_connections 2023-11-15T04:15:50.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.400+0000 7ff85b5df700 1 -- 172.21.15.188:0/2962101948 wait complete. 2023-11-15T04:15:50.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85b5df700 1 Processor -- start 2023-11-15T04:15:50.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85b5df700 1 -- start start 2023-11-15T04:15:50.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85b5df700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85b5df700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854106c20 con 0x7ff8540f6510 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85937b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85937b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56454/0 (socket says 172.21.15.188:56454) 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85937b700 1 -- 172.21.15.188:0/509001598 learned_addr learned my addr 172.21.15.188:0/509001598 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.401+0000 7ff85937b700 1 -- 172.21.15.188:0/509001598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844005040 con 0x7ff8540f6510 2023-11-15T04:15:50.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff85937b700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff844013510 tx=0x7ff84400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff844009320 con 0x7ff8540f6510 2023-11-15T04:15:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff844013f00 con 0x7ff8540f6510 2023-11-15T04:15:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff854106e20 con 0x7ff8540f6510 2023-11-15T04:15:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff844014d90 con 0x7ff8540f6510 2023-11-15T04:15:50.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.402+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8541072b0 con 0x7ff8540f6510 2023-11-15T04:15:50.407 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.403+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8440095f0 con 0x7ff8540f6510 2023-11-15T04:15:50.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.403+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff854045ad0 con 0x7ff8540f6510 2023-11-15T04:15:50.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.403+0000 7ff849ffb700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:50.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.403+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff844040fe0 con 0x7ff8540f6510 2023-11-15T04:15:50.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.403+0000 7ff858b7a700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:50.408 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.404+0000 7ff858b7a700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 0x7ff840032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:50.409 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.408+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff844014420 con 0x7ff8540f6510 2023-11-15T04:15:50.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.613+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8540fbf60 con 0x7ff8540f6510 2023-11-15T04:15:50.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.614+0000 7ff849ffb700 1 -- 172.21.15.188:0/509001598 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff844010230 con 0x7ff8540f6510 2023-11-15T04:15:50.615 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:50.615 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:50.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 msgr2=0x7ff840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:50.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 0x7ff840032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).stop 2023-11-15T04:15:50.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 msgr2=0x7ff854106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:50.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff844013510 tx=0x7ff84400b040).stop 2023-11-15T04:15:50.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 shutdown_connections 2023-11-15T04:15:50.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff84002ff60 0x7ff840032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:50.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 --2- 172.21.15.188:0/509001598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8540f6510 0x7ff854106730 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:50.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 >> 172.21.15.188:0/509001598 conn(0x7ff8540f1c70 msgr2=0x7ff8540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:50.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 shutdown_connections 2023-11-15T04:15:50.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:50.615+0000 7ff85b5df700 1 -- 172.21.15.188:0/509001598 wait complete. 2023-11-15T04:15:50.619 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:50.111969+0000 mgr.smithi038.bnbogz (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:50.214008+0000 mon.smithi038 (mon.0) 275 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:15:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:50.215729+0000 mon.smithi038 (mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:15:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:50.614672+0000 mon.smithi038 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.188:0/509001598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:52.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:52.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:52.112384+0000 mgr.smithi038.bnbogz (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:53.244 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- 172.21.15.188:0/1064837818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 msgr2=0x7fca500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:53.244 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 --2- 172.21.15.188:0/1064837818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca500f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fca4c009fb0 tx=0x7fca4c009b30).stop 2023-11-15T04:15:53.244 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- 172.21.15.188:0/1064837818 shutdown_connections 2023-11-15T04:15:53.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 --2- 172.21.15.188:0/1064837818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca500f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:53.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- 172.21.15.188:0/1064837818 >> 172.21.15.188:0/1064837818 conn(0x7fca500f0b70 msgr2=0x7fca500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:53.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- 172.21.15.188:0/1064837818 shutdown_connections 2023-11-15T04:15:53.245 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- 172.21.15.188:0/1064837818 wait complete. 2023-11-15T04:15:53.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 Processor -- start 2023-11-15T04:15:53.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.242+0000 7fca585db700 1 -- start start 2023-11-15T04:15:53.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca585db700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:53.246 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca585db700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca50060280 con 0x7fca500f5410 2023-11-15T04:15:53.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca56377700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:53.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca56377700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56464/0 (socket says 172.21.15.188:56464) 2023-11-15T04:15:53.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca56377700 1 -- 172.21.15.188:0/3720465959 learned_addr learned my addr 172.21.15.188:0/3720465959 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:53.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.243+0000 7fca56377700 1 -- 172.21.15.188:0/3720465959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c005040 con 0x7fca500f5410 2023-11-15T04:15:53.247 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca56377700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fca4c013510 tx=0x7fca4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:53.249 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca4c009320 con 0x7fca500f5410 2023-11-15T04:15:53.249 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca50060480 con 0x7fca500f5410 2023-11-15T04:15:53.249 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca4c013f00 con 0x7fca500f5410 2023-11-15T04:15:53.249 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca4c014d90 con 0x7fca500f5410 2023-11-15T04:15:53.249 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.244+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca50109ac0 con 0x7fca500f5410 2023-11-15T04:15:53.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.245+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca4c0263a0 con 0x7fca500f5410 2023-11-15T04:15:53.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.245+0000 7fca46ffd700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 0x7fca3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:53.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.245+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca4c0410e0 con 0x7fca500f5410 2023-11-15T04:15:53.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.245+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50045ad0 con 0x7fca500f5410 2023-11-15T04:15:53.250 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.245+0000 7fca55b76700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 0x7fca3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:53.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.246+0000 7fca55b76700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 0x7fca3c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fca400096a0 tx=0x7fca40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:53.251 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.250+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca4c012920 con 0x7fca500f5410 2023-11-15T04:15:53.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.454+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca5010a2a0 con 0x7fca500f5410 2023-11-15T04:15:53.456 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.455+0000 7fca46ffd700 1 -- 172.21.15.188:0/3720465959 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca4c02b020 con 0x7fca500f5410 2023-11-15T04:15:53.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:53.457 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:53.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 msgr2=0x7fca3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:53.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 0x7fca3c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fca400096a0 tx=0x7fca40006b40).stop 2023-11-15T04:15:53.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 msgr2=0x7fca50109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:53.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fca4c013510 tx=0x7fca4c00b040).stop 2023-11-15T04:15:53.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 shutdown_connections 2023-11-15T04:15:53.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fca3c02ff60 0x7fca3c032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:53.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 --2- 172.21.15.188:0/3720465959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca500f5410 0x7fca50109390 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:53.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 >> 172.21.15.188:0/3720465959 conn(0x7fca500f0b70 msgr2=0x7fca500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:53.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 shutdown_connections 2023-11-15T04:15:53.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:53.457+0000 7fca585db700 1 -- 172.21.15.188:0/3720465959 wait complete. 2023-11-15T04:15:53.461 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:53.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:53.456628+0000 mon.smithi038 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.188:0/3720465959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:54.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:54.112772+0000 mgr.smithi038.bnbogz (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:54.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:54.862 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:56.083 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 -- 172.21.15.188:0/2496403835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 msgr2=0x7f8c140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:56.084 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 --2- 172.21.15.188:0/2496403835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c140f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f8c04009fb0 tx=0x7f8c04009b30).stop 2023-11-15T04:15:56.084 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 -- 172.21.15.188:0/2496403835 shutdown_connections 2023-11-15T04:15:56.084 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 --2- 172.21.15.188:0/2496403835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c140f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:56.084 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 -- 172.21.15.188:0/2496403835 >> 172.21.15.188:0/2496403835 conn(0x7f8c140f0b70 msgr2=0x7f8c140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:56.084 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 -- 172.21.15.188:0/2496403835 shutdown_connections 2023-11-15T04:15:56.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.081+0000 7f8c1b623700 1 -- 172.21.15.188:0/2496403835 wait complete. 2023-11-15T04:15:56.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c1b623700 1 Processor -- start 2023-11-15T04:15:56.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c1b623700 1 -- start start 2023-11-15T04:15:56.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c1b623700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:56.085 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c1b623700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14106bd0 con 0x7f8c14060280 2023-11-15T04:15:56.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c193bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:56.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c193bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56480/0 (socket says 172.21.15.188:56480) 2023-11-15T04:15:56.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c193bf700 1 -- 172.21.15.188:0/1248187277 learned_addr learned my addr 172.21.15.188:0/1248187277 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:56.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c193bf700 1 -- 172.21.15.188:0/1248187277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04005040 con 0x7f8c14060280 2023-11-15T04:15:56.086 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.082+0000 7f8c193bf700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f8c04013510 tx=0x7f8c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:56.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.083+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c04009320 con 0x7f8c14060280 2023-11-15T04:15:56.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.083+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c14106dd0 con 0x7f8c14060280 2023-11-15T04:15:56.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.083+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c04013f00 con 0x7f8c14060280 2023-11-15T04:15:56.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.083+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c04014dc0 con 0x7f8c14060280 2023-11-15T04:15:56.087 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.083+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c14107260 con 0x7f8c14060280 2023-11-15T04:15:56.088 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.084+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c040095f0 con 0x7f8c14060280 2023-11-15T04:15:56.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.084+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14045ad0 con 0x7f8c14060280 2023-11-15T04:15:56.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.084+0000 7f8c09ffb700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:56.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.084+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c040410e0 con 0x7f8c14060280 2023-11-15T04:15:56.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.084+0000 7f8c18bbe700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:56.089 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.085+0000 7f8c18bbe700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 0x7f8c00032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8c100096a0 tx=0x7f8c10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:56.090 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.090+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c04014420 con 0x7f8c14060280 2023-11-15T04:15:56.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.293+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c14108110 con 0x7f8c14060280 2023-11-15T04:15:56.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.293+0000 7f8c09ffb700 1 -- 172.21.15.188:0/1248187277 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c0401c920 con 0x7f8c14060280 2023-11-15T04:15:56.294 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:56.295 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:56.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 msgr2=0x7f8c00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:56.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 0x7f8c00032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8c100096a0 tx=0x7f8c10006b40).stop 2023-11-15T04:15:56.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 msgr2=0x7f8c141066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:56.296 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f8c04013510 tx=0x7f8c0400b040).stop 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 shutdown_connections 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f8c0002ff60 0x7f8c00032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 --2- 172.21.15.188:0/1248187277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c14060280 0x7f8c141066e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 >> 172.21.15.188:0/1248187277 conn(0x7f8c140f0b70 msgr2=0x7f8c140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 shutdown_connections 2023-11-15T04:15:56.297 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:56.295+0000 7f8c1b623700 1 -- 172.21.15.188:0/1248187277 wait complete. 2023-11-15T04:15:56.298 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:56.113175+0000 mgr.smithi038.bnbogz (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:15:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:56.294865+0000 mon.smithi038 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.188:0/1248187277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:15:57.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:15:57.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:15:58.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 -- 172.21.15.188:0/3563856428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 msgr2=0x7fa01c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:58.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 --2- 172.21.15.188:0/3563856428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c0f6930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fa00c009fb0 tx=0x7fa00c009b30).stop 2023-11-15T04:15:58.975 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 -- 172.21.15.188:0/3563856428 shutdown_connections 2023-11-15T04:15:58.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 --2- 172.21.15.188:0/3563856428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c0f6930 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:58.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 -- 172.21.15.188:0/3563856428 >> 172.21.15.188:0/3563856428 conn(0x7fa01c0f1c70 msgr2=0x7fa01c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:58.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 -- 172.21.15.188:0/3563856428 shutdown_connections 2023-11-15T04:15:58.976 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 -- 172.21.15.188:0/3563856428 wait complete. 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.973+0000 7fa02405a700 1 Processor -- start 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa02405a700 1 -- start start 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa02405a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa02405a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c106c20 con 0x7fa01c0f6510 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa021df6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:58.977 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa021df6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55808/0 (socket says 172.21.15.188:55808) 2023-11-15T04:15:58.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa021df6700 1 -- 172.21.15.188:0/1102429955 learned_addr learned my addr 172.21.15.188:0/1102429955 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:15:58.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa021df6700 1 -- 172.21.15.188:0/1102429955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c005040 con 0x7fa01c0f6510 2023-11-15T04:15:58.978 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.974+0000 7fa021df6700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa00c013510 tx=0x7fa00c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:58.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa00c009320 con 0x7fa01c0f6510 2023-11-15T04:15:58.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c106e20 con 0x7fa01c0f6510 2023-11-15T04:15:58.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa00c013f00 con 0x7fa01c0f6510 2023-11-15T04:15:58.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa00c014d90 con 0x7fa01c0f6510 2023-11-15T04:15:58.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1072b0 con 0x7fa01c0f6510 2023-11-15T04:15:58.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.975+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa00c0095f0 con 0x7fa01c0f6510 2023-11-15T04:15:58.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.976+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c045ad0 con 0x7fa01c0f6510 2023-11-15T04:15:58.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.976+0000 7fa0127fc700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:15:58.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.976+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa00c041080 con 0x7fa01c0f6510 2023-11-15T04:15:58.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.976+0000 7fa0215f5700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:15:58.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.976+0000 7fa0215f5700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:15:58.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:58.981+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa00c014420 con 0x7fa01c0f6510 2023-11-15T04:15:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.183+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa01c0fbf60 con 0x7fa01c0f6510 2023-11-15T04:15:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.183+0000 7fa0127fc700 1 -- 172.21.15.188:0/1102429955 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa00c010230 con 0x7fa01c0f6510 2023-11-15T04:15:59.184 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:15:59.185 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:15:59.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 msgr2=0x7fa008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 0x7fa008032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa0180096a0 tx=0x7fa018006b40).stop 2023-11-15T04:15:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 msgr2=0x7fa01c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:15:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa00c013510 tx=0x7fa00c00b040).stop 2023-11-15T04:15:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 shutdown_connections 2023-11-15T04:15:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fa00802ff60 0x7fa008032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 --2- 172.21.15.188:0/1102429955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa01c0f6510 0x7fa01c106730 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:15:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 >> 172.21.15.188:0/1102429955 conn(0x7fa01c0f1c70 msgr2=0x7fa01c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:15:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 shutdown_connections 2023-11-15T04:15:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:15:59.185+0000 7fa02405a700 1 -- 172.21.15.188:0/1102429955 wait complete. 2023-11-15T04:15:59.188 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:15:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:15:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:15:58.113620+0000 mgr.smithi038.bnbogz (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:00.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:00 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:15:59.184654+0000 mon.smithi038 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.188:0/1102429955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:00.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:00.610 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:00.114018+0000 mgr.smithi038.bnbogz (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:01.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.831+0000 7f4672560700 1 -- 172.21.15.188:0/589913105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 msgr2=0x7f466c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.831+0000 7f4672560700 1 --2- 172.21.15.188:0/589913105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c0f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f465c009fb0 tx=0x7f465c009b30).stop 2023-11-15T04:16:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.831+0000 7f4672560700 1 -- 172.21.15.188:0/589913105 shutdown_connections 2023-11-15T04:16:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.831+0000 7f4672560700 1 --2- 172.21.15.188:0/589913105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c0f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.831+0000 7f4672560700 1 -- 172.21.15.188:0/589913105 >> 172.21.15.188:0/589913105 conn(0x7f466c0f0b70 msgr2=0x7f466c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 -- 172.21.15.188:0/589913105 shutdown_connections 2023-11-15T04:16:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 -- 172.21.15.188:0/589913105 wait complete. 2023-11-15T04:16:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 Processor -- start 2023-11-15T04:16:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 -- start start 2023-11-15T04:16:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f4672560700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c105ac0 con 0x7f466c0f5410 2023-11-15T04:16:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f466bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f466bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55816/0 (socket says 172.21.15.188:55816) 2023-11-15T04:16:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.832+0000 7f466bfff700 1 -- 172.21.15.188:0/2035230989 learned_addr learned my addr 172.21.15.188:0/2035230989 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f466bfff700 1 -- 172.21.15.188:0/2035230989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c005040 con 0x7f466c0f5410 2023-11-15T04:16:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f466bfff700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f465c014000 tx=0x7f465c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c0095f0 con 0x7f466c0f5410 2023-11-15T04:16:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c105cc0 con 0x7f466c0f5410 2023-11-15T04:16:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f465c007e70 con 0x7f466c0f5410 2023-11-15T04:16:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c014ea0 con 0x7f466c0f5410 2023-11-15T04:16:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.833+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c106150 con 0x7f466c0f5410 2023-11-15T04:16:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.834+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f465c025070 con 0x7f466c0f5410 2023-11-15T04:16:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.834+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c045ad0 con 0x7f466c0f5410 2023-11-15T04:16:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.834+0000 7f46697fa700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 0x7f4658032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.834+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f465c040a00 con 0x7f466c0f5410 2023-11-15T04:16:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.834+0000 7f4663fff700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 0x7f4658032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:01.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.835+0000 7f4663fff700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 0x7f4658032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f46540096a0 tx=0x7f4654006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:01.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:01.839+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f465c0079c0 con 0x7f466c0f5410 2023-11-15T04:16:02.043 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.042+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f466c106fa0 con 0x7f466c0f5410 2023-11-15T04:16:02.044 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.043+0000 7f46697fa700 1 -- 172.21.15.188:0/2035230989 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f465c00f300 con 0x7f466c0f5410 2023-11-15T04:16:02.044 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:02.044 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:02.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 msgr2=0x7f4658032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 0x7f4658032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f46540096a0 tx=0x7f4654006b40).stop 2023-11-15T04:16:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 msgr2=0x7f466c1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f465c014000 tx=0x7f465c00b040).stop 2023-11-15T04:16:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 shutdown_connections 2023-11-15T04:16:02.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f465802ff60 0x7f4658032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:02.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 --2- 172.21.15.188:0/2035230989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c0f5410 0x7f466c1055d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:02.047 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 >> 172.21.15.188:0/2035230989 conn(0x7f466c0f0b70 msgr2=0x7f466c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 shutdown_connections 2023-11-15T04:16:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:02.044+0000 7f4672560700 1 -- 172.21.15.188:0/2035230989 wait complete. 2023-11-15T04:16:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:02.044107+0000 mon.smithi038 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.188:0/2035230989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:02.114366+0000 mgr.smithi038.bnbogz (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:03.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:03.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:04.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 -- 172.21.15.188:0/2978097705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 msgr2=0x7f23740f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:04.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 --2- 172.21.15.188:0/2978097705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f23740f5d70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f2368009fb0 tx=0x7f2368009b30).stop 2023-11-15T04:16:04.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 -- 172.21.15.188:0/2978097705 shutdown_connections 2023-11-15T04:16:04.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 --2- 172.21.15.188:0/2978097705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f23740f5d70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 -- 172.21.15.188:0/2978097705 >> 172.21.15.188:0/2978097705 conn(0x7f23740f0260 msgr2=0x7f23740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 -- 172.21.15.188:0/2978097705 shutdown_connections 2023-11-15T04:16:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 -- 172.21.15.188:0/2978097705 wait complete. 2023-11-15T04:16:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.672+0000 7f2378d7f700 1 Processor -- start 2023-11-15T04:16:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f2378d7f700 1 -- start start 2023-11-15T04:16:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f2378d7f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f2378d7f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23740f5330 con 0x7f23740f5950 2023-11-15T04:16:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f237259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f237259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55836/0 (socket says 172.21.15.188:55836) 2023-11-15T04:16:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.673+0000 7f237259c700 1 -- 172.21.15.188:0/818582304 learned_addr learned my addr 172.21.15.188:0/818582304 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:04.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f237259c700 1 -- 172.21.15.188:0/818582304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368005040 con 0x7f23740f5950 2023-11-15T04:16:04.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f237259c700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2368014000 tx=0x7f236800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:04.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23680095f0 con 0x7f23740f5950 2023-11-15T04:16:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23740f5530 con 0x7f23740f5950 2023-11-15T04:16:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2368007e70 con 0x7f23740f5950 2023-11-15T04:16:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2368014ea0 con 0x7f23740f5950 2023-11-15T04:16:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.674+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2374109ac0 con 0x7f23740f5950 2023-11-15T04:16:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.675+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2368025070 con 0x7f23740f5950 2023-11-15T04:16:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.675+0000 7f2362ffd700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 0x7f235c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.675+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2368040a30 con 0x7f23740f5950 2023-11-15T04:16:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.675+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2374045ad0 con 0x7f23740f5950 2023-11-15T04:16:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.675+0000 7f2371d9b700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 0x7f235c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.676+0000 7f2371d9b700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 0x7f235c032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f23640096a0 tx=0x7f2364006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:04.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.680+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23680079c0 con 0x7f23740f5950 2023-11-15T04:16:04.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.886+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23740fae40 con 0x7f23740f5950 2023-11-15T04:16:04.888 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.887+0000 7f2362ffd700 1 -- 172.21.15.188:0/818582304 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23740fae40 con 0x7f23740f5950 2023-11-15T04:16:04.888 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:04.888 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:04.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 msgr2=0x7f235c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:04.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 0x7f235c032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f23640096a0 tx=0x7f2364006b40).stop 2023-11-15T04:16:04.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 msgr2=0x7f2374109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2368014000 tx=0x7f236800b040).stop 2023-11-15T04:16:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 shutdown_connections 2023-11-15T04:16:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f235c02ff60 0x7f235c032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 --2- 172.21.15.188:0/818582304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23740f5950 0x7f2374109410 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 >> 172.21.15.188:0/818582304 conn(0x7f23740f0260 msgr2=0x7f23740f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 shutdown_connections 2023-11-15T04:16:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:04.888+0000 7f2378d7f700 1 -- 172.21.15.188:0/818582304 wait complete. 2023-11-15T04:16:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:04.114741+0000 mgr.smithi038.bnbogz (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:04.887953+0000 mon.smithi038 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.188:0/818582304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:06.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:06.326 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:07.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- 172.21.15.188:0/2448744622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 msgr2=0x7f17f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:07.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 --2- 172.21.15.188:0/2448744622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f00f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f17e4009fb0 tx=0x7f17e4009b30).stop 2023-11-15T04:16:07.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- 172.21.15.188:0/2448744622 shutdown_connections 2023-11-15T04:16:07.559 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 --2- 172.21.15.188:0/2448744622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f00f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:07.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- 172.21.15.188:0/2448744622 >> 172.21.15.188:0/2448744622 conn(0x7f17f00f1c70 msgr2=0x7f17f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:07.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- 172.21.15.188:0/2448744622 shutdown_connections 2023-11-15T04:16:07.560 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- 172.21.15.188:0/2448744622 wait complete. 2023-11-15T04:16:07.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 Processor -- start 2023-11-15T04:16:07.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.558+0000 7f17f5527700 1 -- start start 2023-11-15T04:16:07.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17f5527700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:07.562 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17f5527700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0069e20 con 0x7f17f00f6510 2023-11-15T04:16:07.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:07.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:55862/0 (socket says 172.21.15.188:55862) 2023-11-15T04:16:07.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17eeffd700 1 -- 172.21.15.188:0/1745579691 learned_addr learned my addr 172.21.15.188:0/1745579691 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:07.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17eeffd700 1 -- 172.21.15.188:0/1745579691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4005040 con 0x7f17f00f6510 2023-11-15T04:16:07.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.559+0000 7f17eeffd700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f17e4013510 tx=0x7f17e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:07.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.560+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4009320 con 0x7f17f00f6510 2023-11-15T04:16:07.564 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.560+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f006a020 con 0x7f17f00f6510 2023-11-15T04:16:07.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.560+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17e4013f00 con 0x7f17f00f6510 2023-11-15T04:16:07.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.560+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4014d90 con 0x7f17f00f6510 2023-11-15T04:16:07.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.560+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f006a4b0 con 0x7f17f00f6510 2023-11-15T04:16:07.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f0059100 con 0x7f17f00f6510 2023-11-15T04:16:07.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17e40095f0 con 0x7f17f00f6510 2023-11-15T04:16:07.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17d77fe700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 0x7f17d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:07.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17e4041000 con 0x7f17f00f6510 2023-11-15T04:16:07.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17ee7fc700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 0x7f17d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:07.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.561+0000 7f17ee7fc700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 0x7f17d8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f17e00096a0 tx=0x7f17e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:07.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.566+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17e4014420 con 0x7f17f00f6510 2023-11-15T04:16:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:06.115100+0000 mgr.smithi038.bnbogz (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:07.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.771+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17f00fbf60 con 0x7f17f00f6510 2023-11-15T04:16:07.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.772+0000 7f17d77fe700 1 -- 172.21.15.188:0/1745579691 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17e401d7f0 con 0x7f17f00f6510 2023-11-15T04:16:07.773 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:07.773 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:07.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 msgr2=0x7f17d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:07.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 0x7f17d8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f17e00096a0 tx=0x7f17e0006b40).stop 2023-11-15T04:16:07.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 msgr2=0x7f17f006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:07.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f17e4013510 tx=0x7f17e400b040).stop 2023-11-15T04:16:07.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 shutdown_connections 2023-11-15T04:16:07.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f17d802ff60 0x7f17d8032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:07.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 --2- 172.21.15.188:0/1745579691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f00f6510 0x7f17f006b930 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:07.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 >> 172.21.15.188:0/1745579691 conn(0x7f17f00f1c70 msgr2=0x7f17f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:07.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 shutdown_connections 2023-11-15T04:16:07.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:07.774+0000 7f17f5527700 1 -- 172.21.15.188:0/1745579691 wait complete. 2023-11-15T04:16:07.777 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:08.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:08 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:07.773555+0000 mon.smithi038 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.188:0/1745579691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:09.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:09.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:08.115471+0000 mgr.smithi038.bnbogz (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:10.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.348+0000 7fe832f46700 1 -- 172.21.15.188:0/2843444393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 msgr2=0x7fe82c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:10.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.348+0000 7fe832f46700 1 --2- 172.21.15.188:0/2843444393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c0f8f80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fe828009fb0 tx=0x7fe828009b30).stop 2023-11-15T04:16:10.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.348+0000 7fe832f46700 1 -- 172.21.15.188:0/2843444393 shutdown_connections 2023-11-15T04:16:10.349 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.348+0000 7fe832f46700 1 --2- 172.21.15.188:0/2843444393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c0f8f80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:10.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.348+0000 7fe832f46700 1 -- 172.21.15.188:0/2843444393 >> 172.21.15.188:0/2843444393 conn(0x7fe82c0f09a0 msgr2=0x7fe82c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:10.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 -- 172.21.15.188:0/2843444393 shutdown_connections 2023-11-15T04:16:10.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 -- 172.21.15.188:0/2843444393 wait complete. 2023-11-15T04:16:10.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 Processor -- start 2023-11-15T04:16:10.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 -- start start 2023-11-15T04:16:10.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:10.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.349+0000 7fe832f46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1069f0 con 0x7fe82c060180 2023-11-15T04:16:10.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.351+0000 7fe831f44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:10.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.351+0000 7fe831f44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47872/0 (socket says 172.21.15.188:47872) 2023-11-15T04:16:10.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.351+0000 7fe831f44700 1 -- 172.21.15.188:0/3113013161 learned_addr learned my addr 172.21.15.188:0/3113013161 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:10.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.353+0000 7fe831f44700 1 -- 172.21.15.188:0/3113013161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828005040 con 0x7fe82c060180 2023-11-15T04:16:10.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.354+0000 7fe831f44700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fe828013510 tx=0x7fe82800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:10.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.355+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe828009320 con 0x7fe82c060180 2023-11-15T04:16:10.355 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.355+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe828013f00 con 0x7fe82c060180 2023-11-15T04:16:10.355 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.355+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c106bf0 con 0x7fe82c060180 2023-11-15T04:16:10.355 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.355+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe828014d90 con 0x7fe82c060180 2023-11-15T04:16:10.355 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.355+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c107080 con 0x7fe82c060180 2023-11-15T04:16:10.356 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.356+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c045ad0 con 0x7fe82c060180 2023-11-15T04:16:10.356 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.356+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8280095f0 con 0x7fe82c060180 2023-11-15T04:16:10.357 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.356+0000 7fe8227fc700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 0x7fe818032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:10.357 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.356+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe828041040 con 0x7fe82c060180 2023-11-15T04:16:10.357 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.356+0000 7fe831743700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 0x7fe818032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:10.357 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.357+0000 7fe831743700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 0x7fe818032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe81c0096a0 tx=0x7fe81c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:10.360 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.361+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe828014420 con 0x7fe82c060180 2023-11-15T04:16:10.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.566+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe82c107f30 con 0x7fe82c060180 2023-11-15T04:16:10.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.567+0000 7fe8227fc700 1 -- 172.21.15.188:0/3113013161 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe82801d7f0 con 0x7fe82c060180 2023-11-15T04:16:10.569 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:10.569 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:10.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 msgr2=0x7fe818032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:10.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 0x7fe818032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe81c0096a0 tx=0x7fe81c006b40).stop 2023-11-15T04:16:10.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 msgr2=0x7fe82c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:10.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fe828013510 tx=0x7fe82800b040).stop 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 shutdown_connections 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fe81802ffb0 0x7fe818032470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 --2- 172.21.15.188:0/3113013161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe82c060180 0x7fe82c106500 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 >> 172.21.15.188:0/3113013161 conn(0x7fe82c0f09a0 msgr2=0x7fe82c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 shutdown_connections 2023-11-15T04:16:10.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:10.569+0000 7fe832f46700 1 -- 172.21.15.188:0/3113013161 wait complete. 2023-11-15T04:16:10.572 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:10.115935+0000 mgr.smithi038.bnbogz (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:10.567461+0000 mon.smithi038 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.188:0/3113013161' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:12.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:12.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:12.116362+0000 mgr.smithi038.bnbogz (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:13.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.164+0000 7f6d70347700 1 -- 172.21.15.188:0/557720037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 msgr2=0x7f6d680f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:13.165 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.164+0000 7f6d70347700 1 --2- 172.21.15.188:0/557720037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d680f4be0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f6d58009fb0 tx=0x7f6d58009b30).stop 2023-11-15T04:16:13.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- 172.21.15.188:0/557720037 shutdown_connections 2023-11-15T04:16:13.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 --2- 172.21.15.188:0/557720037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d680f4be0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:13.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- 172.21.15.188:0/557720037 >> 172.21.15.188:0/557720037 conn(0x7f6d680f02e0 msgr2=0x7f6d680f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:13.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- 172.21.15.188:0/557720037 shutdown_connections 2023-11-15T04:16:13.166 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- 172.21.15.188:0/557720037 wait complete. 2023-11-15T04:16:13.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 Processor -- start 2023-11-15T04:16:13.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- start start 2023-11-15T04:16:13.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:13.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.165+0000 7f6d70347700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68106bf0 con 0x7f6d680f47c0 2023-11-15T04:16:13.167 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d6e0e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:13.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d6e0e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47896/0 (socket says 172.21.15.188:47896) 2023-11-15T04:16:13.168 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d6e0e3700 1 -- 172.21.15.188:0/3591435457 learned_addr learned my addr 172.21.15.188:0/3591435457 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:13.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d6e0e3700 1 -- 172.21.15.188:0/3591435457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58005040 con 0x7f6d680f47c0 2023-11-15T04:16:13.169 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d6e0e3700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6d58013510 tx=0x7f6d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:13.170 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d58009320 con 0x7f6d680f47c0 2023-11-15T04:16:13.170 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d68106df0 con 0x7f6d680f47c0 2023-11-15T04:16:13.170 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d58013f00 con 0x7f6d680f47c0 2023-11-15T04:16:13.170 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d58014d90 con 0x7f6d680f47c0 2023-11-15T04:16:13.171 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.166+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d68107280 con 0x7f6d680f47c0 2023-11-15T04:16:13.171 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.167+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d580095f0 con 0x7f6d680f47c0 2023-11-15T04:16:13.172 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.167+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68045ad0 con 0x7f6d680f47c0 2023-11-15T04:16:13.172 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.167+0000 7f6d5effd700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 0x7f6d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:13.172 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.167+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d58041070 con 0x7f6d680f47c0 2023-11-15T04:16:13.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.167+0000 7f6d6d8e2700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 0x7f6d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:13.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.168+0000 7f6d6d8e2700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 0x7f6d54032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6d640096a0 tx=0x7f6d64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:13.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.173+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d58014420 con 0x7f6d680f47c0 2023-11-15T04:16:13.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.382+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d681080b0 con 0x7f6d680f47c0 2023-11-15T04:16:13.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.383+0000 7f6d5effd700 1 -- 172.21.15.188:0/3591435457 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d58010230 con 0x7f6d680f47c0 2023-11-15T04:16:13.387 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:13.387 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:13.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 msgr2=0x7f6d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 0x7f6d54032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6d640096a0 tx=0x7f6d64006b40).stop 2023-11-15T04:16:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 msgr2=0x7f6d68106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6d58013510 tx=0x7f6d5800b040).stop 2023-11-15T04:16:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 shutdown_connections 2023-11-15T04:16:13.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6d5402ff60 0x7f6d54032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:13.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 --2- 172.21.15.188:0/3591435457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d680f47c0 0x7f6d68106700 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:13.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 >> 172.21.15.188:0/3591435457 conn(0x7f6d680f02e0 msgr2=0x7f6d680f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:13.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 shutdown_connections 2023-11-15T04:16:13.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:13.384+0000 7f6d70347700 1 -- 172.21.15.188:0/3591435457 wait complete. 2023-11-15T04:16:13.390 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:13.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:13.383285+0000 mon.smithi038 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.188:0/3591435457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:14.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:14.763 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:14.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:14.116768+0000 mgr.smithi038.bnbogz (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.068+0000 7ff70a794700 1 -- 172.21.15.188:0/241524969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 msgr2=0x7ff7040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.068+0000 7ff70a794700 1 --2- 172.21.15.188:0/241524969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff7040f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ff6f4009fb0 tx=0x7ff6f4009b30).stop 2023-11-15T04:16:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.068+0000 7ff70a794700 1 -- 172.21.15.188:0/241524969 shutdown_connections 2023-11-15T04:16:16.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.068+0000 7ff70a794700 1 --2- 172.21.15.188:0/241524969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff7040f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:16.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.068+0000 7ff70a794700 1 -- 172.21.15.188:0/241524969 >> 172.21.15.188:0/241524969 conn(0x7ff7040f0b70 msgr2=0x7ff7040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:16.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 -- 172.21.15.188:0/241524969 shutdown_connections 2023-11-15T04:16:16.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 -- 172.21.15.188:0/241524969 wait complete. 2023-11-15T04:16:16.071 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 Processor -- start 2023-11-15T04:16:16.072 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 -- start start 2023-11-15T04:16:16.072 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:16.072 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.069+0000 7ff70a794700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704107240 con 0x7ff7040f5410 2023-11-15T04:16:16.072 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff703fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:16.072 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff703fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:47916/0 (socket says 172.21.15.188:47916) 2023-11-15T04:16:16.073 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff703fff700 1 -- 172.21.15.188:0/3142492701 learned_addr learned my addr 172.21.15.188:0/3142492701 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:16.073 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff703fff700 1 -- 172.21.15.188:0/3142492701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4005040 con 0x7ff7040f5410 2023-11-15T04:16:16.073 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff703fff700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff6f4013510 tx=0x7ff6f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:16.074 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f4009320 con 0x7ff7040f5410 2023-11-15T04:16:16.075 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff704060220 con 0x7ff7040f5410 2023-11-15T04:16:16.075 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6f4013f00 con 0x7ff7040f5410 2023-11-15T04:16:16.075 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f4014d90 con 0x7ff7040f5410 2023-11-15T04:16:16.075 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.070+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70410bd40 con 0x7ff7040f5410 2023-11-15T04:16:16.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.071+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6f40095f0 con 0x7ff7040f5410 2023-11-15T04:16:16.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.071+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff704045ad0 con 0x7ff7040f5410 2023-11-15T04:16:16.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.072+0000 7ff700ff9700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 0x7ff6f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:16.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.072+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6f4041020 con 0x7ff7040f5410 2023-11-15T04:16:16.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.072+0000 7ff7037fe700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 0x7ff6f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:16.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.072+0000 7ff7037fe700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 0x7ff6f0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff6ec0096a0 tx=0x7ff6ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:16.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.077+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6f4014420 con 0x7ff7040f5410 2023-11-15T04:16:16.279 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.280+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7040fae60 con 0x7ff7040f5410 2023-11-15T04:16:16.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.280+0000 7ff700ff9700 1 -- 172.21.15.188:0/3142492701 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6f401d7f0 con 0x7ff7040f5410 2023-11-15T04:16:16.280 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:16.281 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:16.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 msgr2=0x7ff6f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:16.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 0x7ff6f0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff6ec0096a0 tx=0x7ff6ec006b40).stop 2023-11-15T04:16:16.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 msgr2=0x7ff70410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:16.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff6f4013510 tx=0x7ff6f400b040).stop 2023-11-15T04:16:16.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 shutdown_connections 2023-11-15T04:16:16.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff6f002ff60 0x7ff6f0032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:16.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 --2- 172.21.15.188:0/3142492701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7040f5410 0x7ff70410b850 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:16.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 >> 172.21.15.188:0/3142492701 conn(0x7ff7040f0b70 msgr2=0x7ff7040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:16.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 shutdown_connections 2023-11-15T04:16:16.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:16.282+0000 7ff70a794700 1 -- 172.21.15.188:0/3142492701 wait complete. 2023-11-15T04:16:16.284 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:16.117190+0000 mgr.smithi038.bnbogz (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:16.280792+0000 mon.smithi038 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.188:0/3142492701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:17.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:17.678 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:18.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 -- 172.21.15.188:0/4044857723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 msgr2=0x7f6b600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:18.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 --2- 172.21.15.188:0/4044857723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b600f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f6b50009fb0 tx=0x7f6b50009b30).stop 2023-11-15T04:16:18.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 -- 172.21.15.188:0/4044857723 shutdown_connections 2023-11-15T04:16:18.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 --2- 172.21.15.188:0/4044857723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b600f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 -- 172.21.15.188:0/4044857723 >> 172.21.15.188:0/4044857723 conn(0x7f6b600f0b70 msgr2=0x7f6b600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 -- 172.21.15.188:0/4044857723 shutdown_connections 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.904+0000 7f6b66706700 1 -- 172.21.15.188:0/4044857723 wait complete. 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.905+0000 7f6b66706700 1 Processor -- start 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.905+0000 7f6b66706700 1 -- start start 2023-11-15T04:16:18.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.905+0000 7f6b66706700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:18.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.905+0000 7f6b66706700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b60060220 con 0x7f6b600f5410 2023-11-15T04:16:18.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:18.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58008/0 (socket says 172.21.15.188:58008) 2023-11-15T04:16:18.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5ffff700 1 -- 172.21.15.188:0/3601154540 learned_addr learned my addr 172.21.15.188:0/3601154540 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:18.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5ffff700 1 -- 172.21.15.188:0/3601154540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b50005040 con 0x7f6b600f5410 2023-11-15T04:16:18.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5ffff700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6b50013510 tx=0x7f6b5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:18.911 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b50009320 con 0x7f6b600f5410 2023-11-15T04:16:18.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b60060420 con 0x7f6b600f5410 2023-11-15T04:16:18.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b50013f00 con 0x7f6b600f5410 2023-11-15T04:16:18.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b50014d90 con 0x7f6b600f5410 2023-11-15T04:16:18.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.906+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b60109ad0 con 0x7f6b600f5410 2023-11-15T04:16:18.912 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.907+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b500095f0 con 0x7f6b600f5410 2023-11-15T04:16:18.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.907+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b60059100 con 0x7f6b600f5410 2023-11-15T04:16:18.913 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.907+0000 7f6b5cff9700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 0x7f6b48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:18.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.907+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b500410a0 con 0x7f6b600f5410 2023-11-15T04:16:18.914 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.908+0000 7f6b5f7fe700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 0x7f6b48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:18.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.908+0000 7f6b5f7fe700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 0x7f6b48032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6b540096a0 tx=0x7f6b54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:18.915 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:18.912+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b5006d2b0 con 0x7f6b600f5410 2023-11-15T04:16:19.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.117+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b60045ad0 con 0x7f6b600f5410 2023-11-15T04:16:19.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.118+0000 7f6b5cff9700 1 -- 172.21.15.188:0/3601154540 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b5001d7f0 con 0x7f6b600f5410 2023-11-15T04:16:19.120 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:19.120 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:19.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 msgr2=0x7f6b48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:19.121 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 0x7f6b48032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6b540096a0 tx=0x7f6b54006b40).stop 2023-11-15T04:16:19.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 msgr2=0x7f6b60109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:19.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6b50013510 tx=0x7f6b5000b040).stop 2023-11-15T04:16:19.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 shutdown_connections 2023-11-15T04:16:19.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f6b4802ffb0 0x7f6b48032470 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:19.122 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 --2- 172.21.15.188:0/3601154540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b600f5410 0x7f6b60109420 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:19.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.119+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 >> 172.21.15.188:0/3601154540 conn(0x7f6b600f0b70 msgr2=0x7f6b600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:19.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.120+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 shutdown_connections 2023-11-15T04:16:19.123 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:19.120+0000 7f6b66706700 1 -- 172.21.15.188:0/3601154540 wait complete. 2023-11-15T04:16:19.124 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:19.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:18.117551+0000 mgr.smithi038.bnbogz (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:19.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:19.119160+0000 mon.smithi038 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.188:0/3601154540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:20.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:20.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:20.118005+0000 mgr.smithi038.bnbogz (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:21.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.861+0000 7f0bb7298700 1 -- 172.21.15.188:0/1808535777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb00f5410 msgr2=0x7f0bb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:21.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.861+0000 7f0bb7298700 1 --2- 172.21.15.188:0/1808535777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb00f5410 0x7f0bb00f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0ba4009fb0 tx=0x7f0ba4009b30).stop 2023-11-15T04:16:21.864 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.861+0000 7f0bb7298700 1 -- 172.21.15.188:0/1808535777 shutdown_connections 2023-11-15T04:16:21.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.861+0000 7f0bb7298700 1 --2- 172.21.15.188:0/1808535777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb00f5410 0x7f0bb00f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:21.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.861+0000 7f0bb7298700 1 -- 172.21.15.188:0/1808535777 >> 172.21.15.188:0/1808535777 conn(0x7f0bb00f0b70 msgr2=0x7f0bb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:21.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 -- 172.21.15.188:0/1808535777 shutdown_connections 2023-11-15T04:16:21.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 -- 172.21.15.188:0/1808535777 wait complete. 2023-11-15T04:16:21.865 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 Processor -- start 2023-11-15T04:16:21.866 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 -- start start 2023-11-15T04:16:21.866 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:21.866 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb7298700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0106030 con 0x7f0bb0105720 2023-11-15T04:16:21.866 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb5034700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:21.866 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb5034700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58026/0 (socket says 172.21.15.188:58026) 2023-11-15T04:16:21.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.862+0000 7f0bb5034700 1 -- 172.21.15.188:0/4230798597 learned_addr learned my addr 172.21.15.188:0/4230798597 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:21.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0bb5034700 1 -- 172.21.15.188:0/4230798597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4005040 con 0x7f0bb0105720 2023-11-15T04:16:21.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0bb5034700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0ba4009f80 tx=0x7f0ba4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:21.867 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba40136c0 con 0x7f0bb0105720 2023-11-15T04:16:21.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb0106230 con 0x7f0bb0105720 2023-11-15T04:16:21.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ba40095f0 con 0x7f0bb0105720 2023-11-15T04:16:21.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba4014420 con 0x7f0bb0105720 2023-11-15T04:16:21.868 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.863+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb0106d80 con 0x7f0bb0105720 2023-11-15T04:16:21.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.864+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb0059100 con 0x7f0bb0105720 2023-11-15T04:16:21.869 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.864+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ba4025060 con 0x7f0bb0105720 2023-11-15T04:16:21.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.864+0000 7f0ba1ffb700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 0x7f0b9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:21.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.864+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ba4040d80 con 0x7f0bb0105720 2023-11-15T04:16:21.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.865+0000 7f0bb4833700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 0x7f0b9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:21.870 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.865+0000 7f0bb4833700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 0x7f0b9c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0bac0096a0 tx=0x7f0bac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:21.871 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:21.869+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ba401cb60 con 0x7f0bb0105720 2023-11-15T04:16:22.073 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.073+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bb0107540 con 0x7f0bb0105720 2023-11-15T04:16:22.074 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.074+0000 7f0ba1ffb700 1 -- 172.21.15.188:0/4230798597 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ba4042070 con 0x7f0bb0105720 2023-11-15T04:16:22.075 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:22.075 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:22.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 msgr2=0x7f0b9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:22.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 0x7f0b9c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0bac0096a0 tx=0x7f0bac006b40).stop 2023-11-15T04:16:22.076 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 msgr2=0x7f0bb0105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:22.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0ba4009f80 tx=0x7f0ba4009b30).stop 2023-11-15T04:16:22.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 shutdown_connections 2023-11-15T04:16:22.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0b9c02ff60 0x7f0b9c032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:22.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 --2- 172.21.15.188:0/4230798597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bb0105720 0x7f0bb0105b40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:22.077 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 >> 172.21.15.188:0/4230798597 conn(0x7f0bb00f0b70 msgr2=0x7f0bb00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:22.078 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 shutdown_connections 2023-11-15T04:16:22.078 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:22.075+0000 7f0bb7298700 1 -- 172.21.15.188:0/4230798597 wait complete. 2023-11-15T04:16:22.078 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:22.074999+0000 mon.smithi038 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.188:0/4230798597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:22.118320+0000 mgr.smithi038.bnbogz (mgr.14164) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:23.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:23.460 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:24.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 -- 172.21.15.188:0/2135980007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 msgr2=0x7f2bc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:24.800 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 --2- 172.21.15.188:0/2135980007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc00f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2bb4009fb0 tx=0x7f2bb4009b30).stop 2023-11-15T04:16:24.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 -- 172.21.15.188:0/2135980007 shutdown_connections 2023-11-15T04:16:24.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 --2- 172.21.15.188:0/2135980007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc00f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:24.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 -- 172.21.15.188:0/2135980007 >> 172.21.15.188:0/2135980007 conn(0x7f2bc00f1c70 msgr2=0x7f2bc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:24.801 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 -- 172.21.15.188:0/2135980007 shutdown_connections 2023-11-15T04:16:24.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.798+0000 7f2bc5556700 1 -- 172.21.15.188:0/2135980007 wait complete. 2023-11-15T04:16:24.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bc5556700 1 Processor -- start 2023-11-15T04:16:24.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bc5556700 1 -- start start 2023-11-15T04:16:24.802 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bc5556700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:24.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bc5556700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0106c20 con 0x7f2bc00f6510 2023-11-15T04:16:24.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bbeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:24.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bbeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58054/0 (socket says 172.21.15.188:58054) 2023-11-15T04:16:24.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.799+0000 7f2bbeffd700 1 -- 172.21.15.188:0/123060479 learned_addr learned my addr 172.21.15.188:0/123060479 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:24.803 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2bbeffd700 1 -- 172.21.15.188:0/123060479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4005040 con 0x7f2bc00f6510 2023-11-15T04:16:24.804 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2bbeffd700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2bb4013510 tx=0x7f2bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:24.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4009320 con 0x7f2bc00f6510 2023-11-15T04:16:24.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc0106e20 con 0x7f2bc00f6510 2023-11-15T04:16:24.805 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc01072b0 con 0x7f2bc00f6510 2023-11-15T04:16:24.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb4013f00 con 0x7f2bc00f6510 2023-11-15T04:16:24.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.800+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4014d60 con 0x7f2bc00f6510 2023-11-15T04:16:24.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.801+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc0045ad0 con 0x7f2bc00f6510 2023-11-15T04:16:24.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.801+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb40095f0 con 0x7f2bc00f6510 2023-11-15T04:16:24.806 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.801+0000 7f2ba77fe700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 0x7f2ba8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:24.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.801+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb4040ff0 con 0x7f2bc00f6510 2023-11-15T04:16:24.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.801+0000 7f2bbe7fc700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 0x7f2ba8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:24.807 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.802+0000 7f2bbe7fc700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 0x7f2ba8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2bb00096a0 tx=0x7f2bb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:24.808 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:24.806+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb4014420 con 0x7f2bc00f6510 2023-11-15T04:16:25.011 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.010+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bc00fbf60 con 0x7f2bc00f6510 2023-11-15T04:16:25.012 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.011+0000 7f2ba77fe700 1 -- 172.21.15.188:0/123060479 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb4012980 con 0x7f2bc00f6510 2023-11-15T04:16:25.012 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:25.012 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:25.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 msgr2=0x7f2ba8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:25.013 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 0x7f2ba8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2bb00096a0 tx=0x7f2bb0006b40).stop 2023-11-15T04:16:25.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 msgr2=0x7f2bc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:25.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2bb4013510 tx=0x7f2bb400b040).stop 2023-11-15T04:16:25.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 shutdown_connections 2023-11-15T04:16:25.014 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2ba802ff60 0x7f2ba8032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:25.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 --2- 172.21.15.188:0/123060479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc00f6510 0x7f2bc0106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:25.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 >> 172.21.15.188:0/123060479 conn(0x7f2bc00f1c70 msgr2=0x7f2bc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:25.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 shutdown_connections 2023-11-15T04:16:25.015 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:25.012+0000 7f2bc5556700 1 -- 172.21.15.188:0/123060479 wait complete. 2023-11-15T04:16:25.016 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:24.118734+0000 mgr.smithi038.bnbogz (mgr.14164) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:25.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:25.012314+0000 mon.smithi038 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.188:0/123060479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:26.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:26.393 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:27.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:27 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:26.119056+0000 mgr.smithi038.bnbogz (mgr.14164) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:27.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 -- 172.21.15.188:0/2673552229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 msgr2=0x7fd6080f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:27.608 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/2673552229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6080f9150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd600009fb0 tx=0x7fd600009b30).stop 2023-11-15T04:16:27.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 -- 172.21.15.188:0/2673552229 shutdown_connections 2023-11-15T04:16:27.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/2673552229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6080f9150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:27.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 -- 172.21.15.188:0/2673552229 >> 172.21.15.188:0/2673552229 conn(0x7fd6080f0b70 msgr2=0x7fd6080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:27.609 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 -- 172.21.15.188:0/2673552229 shutdown_connections 2023-11-15T04:16:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 -- 172.21.15.188:0/2673552229 wait complete. 2023-11-15T04:16:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.607+0000 7fd60e1c4700 1 Processor -- start 2023-11-15T04:16:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd60e1c4700 1 -- start start 2023-11-15T04:16:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd60e1c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:27.610 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd60e1c4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd608106bd0 con 0x7fd608060220 2023-11-15T04:16:27.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd6077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:27.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd6077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:58070/0 (socket says 172.21.15.188:58070) 2023-11-15T04:16:27.612 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd6077fe700 1 -- 172.21.15.188:0/4133634982 learned_addr learned my addr 172.21.15.188:0/4133634982 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:27.613 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd6077fe700 1 -- 172.21.15.188:0/4133634982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd600005040 con 0x7fd608060220 2023-11-15T04:16:27.613 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.608+0000 7fd6077fe700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd600013510 tx=0x7fd60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:27.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd600009320 con 0x7fd608060220 2023-11-15T04:16:27.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd608106dd0 con 0x7fd608060220 2023-11-15T04:16:27.614 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd600013f00 con 0x7fd608060220 2023-11-15T04:16:27.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd600014dc0 con 0x7fd608060220 2023-11-15T04:16:27.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd608107260 con 0x7fd608060220 2023-11-15T04:16:27.615 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6000095f0 con 0x7fd608060220 2023-11-15T04:16:27.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.609+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608045ad0 con 0x7fd608060220 2023-11-15T04:16:27.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.610+0000 7fd5fffff700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:27.616 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.610+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd600041000 con 0x7fd608060220 2023-11-15T04:16:27.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.610+0000 7fd606ffd700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:27.617 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.610+0000 7fd606ffd700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 0x7fd5e8032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd5f80089e0 tx=0x7fd5f8006ec0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:27.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.615+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd600014420 con 0x7fd608060220 2023-11-15T04:16:27.822 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.821+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd608108110 con 0x7fd608060220 2023-11-15T04:16:27.822 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.822+0000 7fd5fffff700 1 -- 172.21.15.188:0/4133634982 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd600010230 con 0x7fd608060220 2023-11-15T04:16:27.823 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:27.823 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:27.824 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 msgr2=0x7fd5e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:27.825 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 0x7fd5e8032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd5f80089e0 tx=0x7fd5f8006ec0).stop 2023-11-15T04:16:27.825 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 msgr2=0x7fd6081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:27.825 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd600013510 tx=0x7fd60000b040).stop 2023-11-15T04:16:27.825 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 shutdown_connections 2023-11-15T04:16:27.825 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:27.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 --2- 172.21.15.188:0/4133634982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd608060220 0x7fd6081066e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:27.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 >> 172.21.15.188:0/4133634982 conn(0x7fd6080f0b70 msgr2=0x7fd6080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:27.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 shutdown_connections 2023-11-15T04:16:27.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:27.823+0000 7fd60e1c4700 1 -- 172.21.15.188:0/4133634982 wait complete. 2023-11-15T04:16:27.826 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:28.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:28 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:27.823108+0000 mon.smithi038 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.188:0/4133634982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:29.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:29.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:28.119343+0000 mgr.smithi038.bnbogz (mgr.14164) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:30.378 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1493784064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 msgr2=0x7f7fcc0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:30.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1493784064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f9190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f7fb4009fb0 tx=0x7f7fb4009b30).stop 2023-11-15T04:16:30.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1493784064 shutdown_connections 2023-11-15T04:16:30.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1493784064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f9190 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:30.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1493784064 >> 172.21.15.188:0/1493784064 conn(0x7f7fcc0f0b70 msgr2=0x7f7fcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:30.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1493784064 shutdown_connections 2023-11-15T04:16:30.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1493784064 wait complete. 2023-11-15T04:16:30.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.377+0000 7f7fd1c92700 1 Processor -- start 2023-11-15T04:16:30.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fd1c92700 1 -- start start 2023-11-15T04:16:30.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fd1c92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:30.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fd1c92700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0f5090 con 0x7f7fcc0601e0 2023-11-15T04:16:30.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:30.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36062/0 (socket says 172.21.15.188:36062) 2023-11-15T04:16:30.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fcb7fe700 1 -- 172.21.15.188:0/1458761762 learned_addr learned my addr 172.21.15.188:0/1458761762 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:30.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fcb7fe700 1 -- 172.21.15.188:0/1458761762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb4005040 con 0x7f7fcc0601e0 2023-11-15T04:16:30.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.378+0000 7f7fcb7fe700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7fb4004dd0 tx=0x7f7fb4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:30.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fb4018070 con 0x7f7fcc0601e0 2023-11-15T04:16:30.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc0f5230 con 0x7f7fcc0601e0 2023-11-15T04:16:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fb4013410 con 0x7f7fcc0601e0 2023-11-15T04:16:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fb401c3c0 con 0x7f7fcc0601e0 2023-11-15T04:16:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc0f56c0 con 0x7f7fcc0601e0 2023-11-15T04:16:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fb401d420 con 0x7f7fcc0601e0 2023-11-15T04:16:30.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.379+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc059100 con 0x7f7fcc0601e0 2023-11-15T04:16:30.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.380+0000 7f7fc3fff700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 0x7f7fb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:30.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.380+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fb40413f0 con 0x7f7fcc0601e0 2023-11-15T04:16:30.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.380+0000 7f7fcaffd700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 0x7f7fb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:30.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.380+0000 7f7fcaffd700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 0x7f7fb8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7fbc0096a0 tx=0x7f7fbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:30.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.384+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fb401d820 con 0x7f7fcc0601e0 2023-11-15T04:16:30.591 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.590+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fcc0fbf60 con 0x7f7fcc0601e0 2023-11-15T04:16:30.592 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.591+0000 7f7fc3fff700 1 -- 172.21.15.188:0/1458761762 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fb401b370 con 0x7f7fcc0601e0 2023-11-15T04:16:30.592 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:30.592 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:30.594 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 msgr2=0x7f7fb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:30.594 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 0x7f7fb8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7fbc0096a0 tx=0x7f7fbc006b40).stop 2023-11-15T04:16:30.594 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 msgr2=0x7f7fcc0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:30.594 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7fb4004dd0 tx=0x7f7fb4009b30).stop 2023-11-15T04:16:30.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 shutdown_connections 2023-11-15T04:16:30.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7fb802ff60 0x7f7fb8032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:30.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 --2- 172.21.15.188:0/1458761762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fcc0601e0 0x7f7fcc0f6c10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:30.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 >> 172.21.15.188:0/1458761762 conn(0x7f7fcc0f0b70 msgr2=0x7f7fcc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:30.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 shutdown_connections 2023-11-15T04:16:30.596 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:30.593+0000 7f7fd1c92700 1 -- 172.21.15.188:0/1458761762 wait complete. 2023-11-15T04:16:30.596 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:30.119688+0000 mgr.smithi038.bnbogz (mgr.14164) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:30.592731+0000 mon.smithi038 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.188:0/1458761762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:32.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:32.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:32.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:32.120027+0000 mgr.smithi038.bnbogz (mgr.14164) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:33.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 -- 172.21.15.188:0/1247761452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 msgr2=0x7f7a180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:33.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/1247761452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a180f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f7a0c009fb0 tx=0x7f7a0c009b30).stop 2023-11-15T04:16:33.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 -- 172.21.15.188:0/1247761452 shutdown_connections 2023-11-15T04:16:33.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/1247761452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a180f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:33.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 -- 172.21.15.188:0/1247761452 >> 172.21.15.188:0/1247761452 conn(0x7f7a180f1c70 msgr2=0x7f7a180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:33.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 -- 172.21.15.188:0/1247761452 shutdown_connections 2023-11-15T04:16:33.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.222+0000 7f7a1d48d700 1 -- 172.21.15.188:0/1247761452 wait complete. 2023-11-15T04:16:33.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a1d48d700 1 Processor -- start 2023-11-15T04:16:33.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a1d48d700 1 -- start start 2023-11-15T04:16:33.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a1d48d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:33.226 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a1d48d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18069e20 con 0x7f7a180f6510 2023-11-15T04:16:33.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a16ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:33.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a16ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36078/0 (socket says 172.21.15.188:36078) 2023-11-15T04:16:33.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.223+0000 7f7a16ffd700 1 -- 172.21.15.188:0/441087054 learned_addr learned my addr 172.21.15.188:0/441087054 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:33.227 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f7a16ffd700 1 -- 172.21.15.188:0/441087054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c005040 con 0x7f7a180f6510 2023-11-15T04:16:33.228 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f7a16ffd700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f7a0c013510 tx=0x7f7a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:33.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a0c009320 con 0x7f7a180f6510 2023-11-15T04:16:33.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18069fc0 con 0x7f7a180f6510 2023-11-15T04:16:33.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a0c013f00 con 0x7f7a180f6510 2023-11-15T04:16:33.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a0c014d90 con 0x7f7a180f6510 2023-11-15T04:16:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.224+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1806a450 con 0x7f7a180f6510 2023-11-15T04:16:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.225+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a0c0095f0 con 0x7f7a180f6510 2023-11-15T04:16:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.225+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a18059100 con 0x7f7a180f6510 2023-11-15T04:16:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.225+0000 7f79ff7fe700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:33.230 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.225+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a0c041080 con 0x7f7a180f6510 2023-11-15T04:16:33.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.225+0000 7f7a167fc700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:33.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.226+0000 7f7a167fc700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 0x7f7a00032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:33.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.230+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a0c014420 con 0x7f7a180f6510 2023-11-15T04:16:33.435 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.434+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a18045ad0 con 0x7f7a180f6510 2023-11-15T04:16:33.435 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.435+0000 7f79ff7fe700 1 -- 172.21.15.188:0/441087054 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a0c010230 con 0x7f7a180f6510 2023-11-15T04:16:33.436 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:33.436 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:33.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 msgr2=0x7f7a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:33.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 0x7f7a00032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).stop 2023-11-15T04:16:33.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 msgr2=0x7f7a1806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:33.437 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f7a0c013510 tx=0x7f7a0c00b040).stop 2023-11-15T04:16:33.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 shutdown_connections 2023-11-15T04:16:33.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:33.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 --2- 172.21.15.188:0/441087054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a180f6510 0x7f7a1806b9a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:33.438 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 >> 172.21.15.188:0/441087054 conn(0x7f7a180f1c70 msgr2=0x7f7a180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:33.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 shutdown_connections 2023-11-15T04:16:33.439 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:33.436+0000 7f7a1d48d700 1 -- 172.21.15.188:0/441087054 wait complete. 2023-11-15T04:16:33.440 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:33.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:33.436070+0000 mon.smithi038 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.188:0/441087054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:34.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:34.817 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:34.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:34.120320+0000 mgr.smithi038.bnbogz (mgr.14164) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:36.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.020+0000 7f4d76a8c700 1 -- 172.21.15.188:0/208263404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 msgr2=0x7f4d700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:36.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.020+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/208263404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d700f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f4d58009fb0 tx=0x7f4d58009b30).stop 2023-11-15T04:16:36.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.020+0000 7f4d76a8c700 1 -- 172.21.15.188:0/208263404 shutdown_connections 2023-11-15T04:16:36.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.020+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/208263404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d700f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:36.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.020+0000 7f4d76a8c700 1 -- 172.21.15.188:0/208263404 >> 172.21.15.188:0/208263404 conn(0x7f4d700f1c70 msgr2=0x7f4d700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:36.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 -- 172.21.15.188:0/208263404 shutdown_connections 2023-11-15T04:16:36.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 -- 172.21.15.188:0/208263404 wait complete. 2023-11-15T04:16:36.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 Processor -- start 2023-11-15T04:16:36.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 -- start start 2023-11-15T04:16:36.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:36.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.021+0000 7f4d76a8c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70106c20 con 0x7f4d700f6510 2023-11-15T04:16:36.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.023+0000 7f4d74828700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:36.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.023+0000 7f4d74828700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36108/0 (socket says 172.21.15.188:36108) 2023-11-15T04:16:36.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.023+0000 7f4d74828700 1 -- 172.21.15.188:0/2082294055 learned_addr learned my addr 172.21.15.188:0/2082294055 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:36.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.024+0000 7f4d74828700 1 -- 172.21.15.188:0/2082294055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58005040 con 0x7f4d700f6510 2023-11-15T04:16:36.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.024+0000 7f4d74828700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4d58013510 tx=0x7f4d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:36.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.024+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d58009320 con 0x7f4d700f6510 2023-11-15T04:16:36.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.024+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d70106e20 con 0x7f4d700f6510 2023-11-15T04:16:36.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.025+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d58013f00 con 0x7f4d700f6510 2023-11-15T04:16:36.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.025+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d58014d90 con 0x7f4d700f6510 2023-11-15T04:16:36.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.025+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d701072b0 con 0x7f4d700f6510 2023-11-15T04:16:36.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.025+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d580095f0 con 0x7f4d700f6510 2023-11-15T04:16:36.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.025+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70045ad0 con 0x7f4d700f6510 2023-11-15T04:16:36.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.026+0000 7f4d6d7fa700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 0x7f4d5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:36.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.026+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d58041050 con 0x7f4d700f6510 2023-11-15T04:16:36.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.026+0000 7f4d6ffff700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 0x7f4d5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:36.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.026+0000 7f4d6ffff700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 0x7f4d5c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4d600096a0 tx=0x7f4d60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:36.031 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.030+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d58014420 con 0x7f4d700f6510 2023-11-15T04:16:36.234 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.233+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d700fbf60 con 0x7f4d700f6510 2023-11-15T04:16:36.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.234+0000 7f4d6d7fa700 1 -- 172.21.15.188:0/2082294055 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d58010230 con 0x7f4d700f6510 2023-11-15T04:16:36.235 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:36.236 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:36.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 msgr2=0x7f4d5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:36.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 0x7f4d5c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4d600096a0 tx=0x7f4d60006b40).stop 2023-11-15T04:16:36.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 msgr2=0x7f4d70106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:36.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4d58013510 tx=0x7f4d5800b040).stop 2023-11-15T04:16:36.237 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 shutdown_connections 2023-11-15T04:16:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f4d5c02ffb0 0x7f4d5c032470 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 --2- 172.21.15.188:0/2082294055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d700f6510 0x7f4d70106730 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 >> 172.21.15.188:0/2082294055 conn(0x7f4d700f1c70 msgr2=0x7f4d700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 shutdown_connections 2023-11-15T04:16:36.238 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:36.236+0000 7f4d76a8c700 1 -- 172.21.15.188:0/2082294055 wait complete. 2023-11-15T04:16:36.239 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:36.120591+0000 mgr.smithi038.bnbogz (mgr.14164) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:36.235739+0000 mon.smithi038 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.188:0/2082294055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:37.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:37.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:38.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 -- 172.21.15.188:0/2876148055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 msgr2=0x7f1e3c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:38.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/2876148055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c0f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f1e2c009fb0 tx=0x7f1e2c009b30).stop 2023-11-15T04:16:38.959 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 -- 172.21.15.188:0/2876148055 shutdown_connections 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/2876148055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c0f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 -- 172.21.15.188:0/2876148055 >> 172.21.15.188:0/2876148055 conn(0x7f1e3c0f09a0 msgr2=0x7f1e3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 -- 172.21.15.188:0/2876148055 shutdown_connections 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.957+0000 7f1e40f3d700 1 -- 172.21.15.188:0/2876148055 wait complete. 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e40f3d700 1 Processor -- start 2023-11-15T04:16:38.960 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e40f3d700 1 -- start start 2023-11-15T04:16:38.961 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e40f3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:38.961 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e40f3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c1069d0 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.962 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e3b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:38.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e3b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:32876/0 (socket says 172.21.15.188:32876) 2023-11-15T04:16:38.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.958+0000 7f1e3b7fe700 1 -- 172.21.15.188:0/3102712971 learned_addr learned my addr 172.21.15.188:0/3102712971 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:38.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e3b7fe700 1 -- 172.21.15.188:0/3102712971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c005040 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e3b7fe700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1e2c009830 tx=0x7f1e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:38.963 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2c009320 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c106bd0 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c107060 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e2c0136c0 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.959+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2c01ee90 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.964 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.960+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c045ad0 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.960+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e2c01e420 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.960+0000 7f1e23fff700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:38.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.960+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e2c0413d0 con 0x7f1e3c0f51a0 2023-11-15T04:16:38.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.961+0000 7f1e3affd700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:38.965 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.961+0000 7f1e3affd700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 0x7f1e240323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1e300096a0 tx=0x7f1e30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:38.966 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:38.965+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e2c0095f0 con 0x7f1e3c0f51a0 2023-11-15T04:16:39.171 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.170+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e3c107e60 con 0x7f1e3c0f51a0 2023-11-15T04:16:39.172 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.171+0000 7f1e23fff700 1 -- 172.21.15.188:0/3102712971 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e2c01c970 con 0x7f1e3c0f51a0 2023-11-15T04:16:39.173 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:39.173 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:39.175 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 msgr2=0x7f1e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:39.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 0x7f1e240323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1e300096a0 tx=0x7f1e30006b40).stop 2023-11-15T04:16:39.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 msgr2=0x7f1e3c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:39.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1e2c009830 tx=0x7f1e2c00b040).stop 2023-11-15T04:16:39.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 shutdown_connections 2023-11-15T04:16:39.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:39.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 --2- 172.21.15.188:0/3102712971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c0f51a0 0x7f1e3c1064e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:39.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 >> 172.21.15.188:0/3102712971 conn(0x7f1e3c0f09a0 msgr2=0x7f1e3c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:39.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.172+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 shutdown_connections 2023-11-15T04:16:39.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:39.173+0000 7f1e40f3d700 1 -- 172.21.15.188:0/3102712971 wait complete. 2023-11-15T04:16:39.178 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:38.120967+0000 mgr.smithi038.bnbogz (mgr.14164) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:39.172157+0000 mon.smithi038 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.188:0/3102712971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:40.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:40.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:41.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:40.121347+0000 mgr.smithi038.bnbogz (mgr.14164) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:41.885 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 -- 172.21.15.188:0/72196651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 msgr2=0x7f20c80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:41.885 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 --2- 172.21.15.188:0/72196651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c80f9150 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f20b8009fb0 tx=0x7f20b8009b30).stop 2023-11-15T04:16:41.885 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 -- 172.21.15.188:0/72196651 shutdown_connections 2023-11-15T04:16:41.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 --2- 172.21.15.188:0/72196651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c80f9150 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:41.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 -- 172.21.15.188:0/72196651 >> 172.21.15.188:0/72196651 conn(0x7f20c80f0b70 msgr2=0x7f20c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:41.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 -- 172.21.15.188:0/72196651 shutdown_connections 2023-11-15T04:16:41.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.883+0000 7f20d023e700 1 -- 172.21.15.188:0/72196651 wait complete. 2023-11-15T04:16:41.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20d023e700 1 Processor -- start 2023-11-15T04:16:41.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20d023e700 1 -- start start 2023-11-15T04:16:41.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20d023e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:41.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20d023e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8106bd0 con 0x7f20c8060220 2023-11-15T04:16:41.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20cdfda700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:41.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20cdfda700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:32890/0 (socket says 172.21.15.188:32890) 2023-11-15T04:16:41.888 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.884+0000 7f20cdfda700 1 -- 172.21.15.188:0/2377429979 learned_addr learned my addr 172.21.15.188:0/2377429979 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:41.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.885+0000 7f20cdfda700 1 -- 172.21.15.188:0/2377429979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b8005040 con 0x7f20c8060220 2023-11-15T04:16:41.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.885+0000 7f20cdfda700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f20b8009830 tx=0x7f20b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:41.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b8009320 con 0x7f20c8060220 2023-11-15T04:16:41.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c8106dd0 con 0x7f20c8060220 2023-11-15T04:16:41.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20b80136c0 con 0x7f20c8060220 2023-11-15T04:16:41.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b801ee90 con 0x7f20c8060220 2023-11-15T04:16:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c8107260 con 0x7f20c8060220 2023-11-15T04:16:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20b801e420 con 0x7f20c8060220 2023-11-15T04:16:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.886+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c8045ad0 con 0x7f20c8060220 2023-11-15T04:16:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.887+0000 7f20be7fc700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 0x7f20b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:41.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.887+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20b8041010 con 0x7f20c8060220 2023-11-15T04:16:41.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.887+0000 7f20cd7d9700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 0x7f20b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:41.892 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.887+0000 7f20cd7d9700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 0x7f20b40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f20c40096a0 tx=0x7f20c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:41.893 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:41.892+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20b80095f0 con 0x7f20c8060220 2023-11-15T04:16:42.096 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.095+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20c8108110 con 0x7f20c8060220 2023-11-15T04:16:42.096 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.096+0000 7f20be7fc700 1 -- 172.21.15.188:0/2377429979 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20b801cb10 con 0x7f20c8060220 2023-11-15T04:16:42.097 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:42.097 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:42.098 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 msgr2=0x7f20b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:42.098 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 0x7f20b40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f20c40096a0 tx=0x7f20c4006b40).stop 2023-11-15T04:16:42.099 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 msgr2=0x7f20c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:42.099 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f20b8009830 tx=0x7f20b800b040).stop 2023-11-15T04:16:42.099 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 shutdown_connections 2023-11-15T04:16:42.099 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f20b402ff10 0x7f20b40323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:42.099 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 --2- 172.21.15.188:0/2377429979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8060220 0x7f20c81066e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:42.100 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 >> 172.21.15.188:0/2377429979 conn(0x7f20c80f0b70 msgr2=0x7f20c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:42.100 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 shutdown_connections 2023-11-15T04:16:42.100 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:42.097+0000 7f20d023e700 1 -- 172.21.15.188:0/2377429979 wait complete. 2023-11-15T04:16:42.100 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:42.097068+0000 mon.smithi038 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.188:0/2377429979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:42.121685+0000 mgr.smithi038.bnbogz (mgr.14164) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:43.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:43.474 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:44.833 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 -- 172.21.15.188:0/1498073020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 msgr2=0x7f05500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:44.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 --2- 172.21.15.188:0/1498073020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f05500f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f0544009fb0 tx=0x7f0544009b30).stop 2023-11-15T04:16:44.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 -- 172.21.15.188:0/1498073020 shutdown_connections 2023-11-15T04:16:44.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 --2- 172.21.15.188:0/1498073020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f05500f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:44.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 -- 172.21.15.188:0/1498073020 >> 172.21.15.188:0/1498073020 conn(0x7f05500f0b70 msgr2=0x7f05500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:44.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 -- 172.21.15.188:0/1498073020 shutdown_connections 2023-11-15T04:16:44.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.832+0000 7f0555230700 1 -- 172.21.15.188:0/1498073020 wait complete. 2023-11-15T04:16:44.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f0555230700 1 Processor -- start 2023-11-15T04:16:44.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f0555230700 1 -- start start 2023-11-15T04:16:44.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f0555230700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:44.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f0555230700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550105b20 con 0x7f05500f5410 2023-11-15T04:16:44.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f054ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:44.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f054ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:32906/0 (socket says 172.21.15.188:32906) 2023-11-15T04:16:44.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f054ed9d700 1 -- 172.21.15.188:0/3039021209 learned_addr learned my addr 172.21.15.188:0/3039021209 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:44.838 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f054ed9d700 1 -- 172.21.15.188:0/3039021209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544005040 con 0x7f05500f5410 2023-11-15T04:16:44.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.833+0000 7f054ed9d700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f0544013510 tx=0x7f054400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:44.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0544009320 con 0x7f05500f5410 2023-11-15T04:16:44.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0550105d20 con 0x7f05500f5410 2023-11-15T04:16:44.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0544013f00 con 0x7f05500f5410 2023-11-15T04:16:44.839 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0544014d90 con 0x7f05500f5410 2023-11-15T04:16:44.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05501061b0 con 0x7f05500f5410 2023-11-15T04:16:44.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.834+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530004fa0 con 0x7f05500f5410 2023-11-15T04:16:44.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.835+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05440263a0 con 0x7f05500f5410 2023-11-15T04:16:44.840 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.835+0000 7f05377fe700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 0x7f0538032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:44.841 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.835+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0544040fe0 con 0x7f05500f5410 2023-11-15T04:16:44.841 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.835+0000 7f054e59c700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 0x7f0538032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:44.841 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.835+0000 7f054e59c700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 0x7f0538032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:44.842 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:44.841+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05440126f0 con 0x7f05500f5410 2023-11-15T04:16:45.045 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.045+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0530005670 con 0x7f05500f5410 2023-11-15T04:16:45.046 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.046+0000 7f05377fe700 1 -- 172.21.15.188:0/3039021209 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0544012330 con 0x7f05500f5410 2023-11-15T04:16:45.047 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:45.047 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:45.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 msgr2=0x7f0538032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:45.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 0x7f0538032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).stop 2023-11-15T04:16:45.048 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 msgr2=0x7f0550105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:45.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f0544013510 tx=0x7f054400b040).stop 2023-11-15T04:16:45.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 shutdown_connections 2023-11-15T04:16:45.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f053802ff60 0x7f0538032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:45.049 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 --2- 172.21.15.188:0/3039021209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05500f5410 0x7f0550105630 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:45.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 >> 172.21.15.188:0/3039021209 conn(0x7f05500f0b70 msgr2=0x7f05500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:45.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 shutdown_connections 2023-11-15T04:16:45.050 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:45.047+0000 7f0555230700 1 -- 172.21.15.188:0/3039021209 wait complete. 2023-11-15T04:16:45.051 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:44.122151+0000 mgr.smithi038.bnbogz (mgr.14164) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:45.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:45.046995+0000 mon.smithi038 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.188:0/3039021209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:46.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:46.446 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:47.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:47 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:46.122527+0000 mgr.smithi038.bnbogz (mgr.14164) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:47.683 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f941159a700 1 -- 172.21.15.188:0/2682234744 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93fc0138a0 con 0x7f94140f5410 2023-11-15T04:16:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 -- 172.21.15.188:0/2682234744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 msgr2=0x7f94140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 --2- 172.21.15.188:0/2682234744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f94140f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f93fc009fb0 tx=0x7f93fc009b30).stop 2023-11-15T04:16:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 -- 172.21.15.188:0/2682234744 shutdown_connections 2023-11-15T04:16:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 --2- 172.21.15.188:0/2682234744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f94140f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:47.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 -- 172.21.15.188:0/2682234744 >> 172.21.15.188:0/2682234744 conn(0x7f94140f0b70 msgr2=0x7f94140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 -- 172.21.15.188:0/2682234744 shutdown_connections 2023-11-15T04:16:47.685 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.682+0000 7f9418ef9700 1 -- 172.21.15.188:0/2682234744 wait complete. 2023-11-15T04:16:47.686 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.683+0000 7f9418ef9700 1 Processor -- start 2023-11-15T04:16:47.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.683+0000 7f9418ef9700 1 -- start start 2023-11-15T04:16:47.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.683+0000 7f9418ef9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:47.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.683+0000 7f9418ef9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414104a10 con 0x7f94140f5410 2023-11-15T04:16:47.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.684+0000 7f941259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:47.687 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.684+0000 7f941259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:32926/0 (socket says 172.21.15.188:32926) 2023-11-15T04:16:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.684+0000 7f941259c700 1 -- 172.21.15.188:0/3846953512 learned_addr learned my addr 172.21.15.188:0/3846953512 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.684+0000 7f941259c700 1 -- 172.21.15.188:0/3846953512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc005040 con 0x7f94140f5410 2023-11-15T04:16:47.688 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f941259c700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f93fc013510 tx=0x7f93fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93fc009320 con 0x7f94140f5410 2023-11-15T04:16:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93fc013f00 con 0x7f94140f5410 2023-11-15T04:16:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9414104c10 con 0x7f94140f5410 2023-11-15T04:16:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93fc014d90 con 0x7f94140f5410 2023-11-15T04:16:47.689 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.685+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141050a0 con 0x7f94140f5410 2023-11-15T04:16:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.686+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93fc0095f0 con 0x7f94140f5410 2023-11-15T04:16:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.686+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9414059100 con 0x7f94140f5410 2023-11-15T04:16:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.686+0000 7f940affd700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 0x7f9400036820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:47.690 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.686+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93fc041030 con 0x7f94140f5410 2023-11-15T04:16:47.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.686+0000 7f9411d9b700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 0x7f9400036820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:47.691 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.687+0000 7f9411d9b700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 0x7f9400036820 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f94040096a0 tx=0x7f9404006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:47.692 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.691+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93fc014420 con 0x7f94140f5410 2023-11-15T04:16:47.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.897+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94140f9ee0 con 0x7f94140f5410 2023-11-15T04:16:47.898 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.898+0000 7f940affd700 1 -- 172.21.15.188:0/3846953512 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93fc010230 con 0x7f94140f5410 2023-11-15T04:16:47.899 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:47.899 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 msgr2=0x7f9400036820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:47.900 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 0x7f9400036820 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f94040096a0 tx=0x7f9404006b40).stop 2023-11-15T04:16:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 msgr2=0x7f9414104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f93fc013510 tx=0x7f93fc00b040).stop 2023-11-15T04:16:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 shutdown_connections 2023-11-15T04:16:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9400034360 0x7f9400036820 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:47.901 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 --2- 172.21.15.188:0/3846953512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94140f5410 0x7f9414104520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:47.902 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.899+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 >> 172.21.15.188:0/3846953512 conn(0x7f94140f0b70 msgr2=0x7f94140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:47.902 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.900+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 shutdown_connections 2023-11-15T04:16:47.902 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:47.900+0000 7f9418ef9700 1 -- 172.21.15.188:0/3846953512 wait complete. 2023-11-15T04:16:47.903 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:48.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:48 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:47.899095+0000 mon.smithi038 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.188:0/3846953512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:49.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:49.288 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:48.122878+0000 mgr.smithi038.bnbogz (mgr.14164) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:50.565 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.563+0000 7efeb9a54700 1 -- 172.21.15.188:0/770378183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 msgr2=0x7efeb40f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:50.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.563+0000 7efeb9a54700 1 --2- 172.21.15.188:0/770378183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb40f6e00 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7efea8009fb0 tx=0x7efea8009b30).stop 2023-11-15T04:16:50.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- 172.21.15.188:0/770378183 shutdown_connections 2023-11-15T04:16:50.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 --2- 172.21.15.188:0/770378183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb40f6e00 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:50.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- 172.21.15.188:0/770378183 >> 172.21.15.188:0/770378183 conn(0x7efeb40f02e0 msgr2=0x7efeb40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:50.566 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- 172.21.15.188:0/770378183 shutdown_connections 2023-11-15T04:16:50.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- 172.21.15.188:0/770378183 wait complete. 2023-11-15T04:16:50.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 Processor -- start 2023-11-15T04:16:50.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- start start 2023-11-15T04:16:50.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:50.567 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.564+0000 7efeb9a54700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb4105ae0 con 0x7efeb40f69e0 2023-11-15T04:16:50.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.565+0000 7efeb2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:50.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.565+0000 7efeb2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:44154/0 (socket says 172.21.15.188:44154) 2023-11-15T04:16:50.568 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.565+0000 7efeb2ffd700 1 -- 172.21.15.188:0/1383209973 learned_addr learned my addr 172.21.15.188:0/1383209973 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:50.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.565+0000 7efeb2ffd700 1 -- 172.21.15.188:0/1383209973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea8005040 con 0x7efeb40f69e0 2023-11-15T04:16:50.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.565+0000 7efeb2ffd700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7efea8013510 tx=0x7efea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:50.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea8009320 con 0x7efeb40f69e0 2023-11-15T04:16:50.570 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb4105ce0 con 0x7efeb40f69e0 2023-11-15T04:16:50.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efea8013f00 con 0x7efeb40f69e0 2023-11-15T04:16:50.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea8014d90 con 0x7efeb40f69e0 2023-11-15T04:16:50.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb4106170 con 0x7efeb40f69e0 2023-11-15T04:16:50.571 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efea80095f0 con 0x7efeb40f69e0 2023-11-15T04:16:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.566+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4045ad0 con 0x7efeb40f69e0 2023-11-15T04:16:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.567+0000 7efe9b7fe700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 0x7efe9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.567+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efea8041000 con 0x7efeb40f69e0 2023-11-15T04:16:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.567+0000 7efeb27fc700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 0x7efe9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:50.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.567+0000 7efeb27fc700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 0x7efe9c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:50.573 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.572+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efea8014420 con 0x7efeb40f69e0 2023-11-15T04:16:50.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:16:48.885426+0000 mgr.smithi038.bnbogz (mgr.14164) 246 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi038 ... 2023-11-15T04:16:50.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Verifying port 9100 ... 2023-11-15T04:16:50.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 1 from systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038 2023-11-15T04:16:50.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: systemctl: stderr Job for ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038.service failed because a timeout was exceeded. 2023-11-15T04:16:50.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: systemctl: stderr See "systemctl status ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038.service" and "journalctl -xe" for details. 2023-11-15T04:16:50.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:16:50.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:16:50.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: main() 2023-11-15T04:16:50.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:16:50.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: r = ctx.func(ctx) 2023-11-15T04:16:50.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:16:50.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: return func(ctx) 2023-11-15T04:16:50.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T04:16:50.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ports=daemon_ports) 2023-11-15T04:16:50.595 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T04:16:50.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T04:16:50.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T04:16:50.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T04:16:50.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T04:16:50.596 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T04:16:50.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: RuntimeError: Failed command: systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038 2023-11-15T04:16:50.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:16:50.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T04:16:50.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: yield (conn, connr) 2023-11-15T04:16:50.597 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T04:16:50.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: code, '\n'.join(err))) 2023-11-15T04:16:50.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi038 ... 2023-11-15T04:16:50.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Verifying port 9100 ... 2023-11-15T04:16:50.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Non-zero exit code 1 from systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038 2023-11-15T04:16:50.598 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: systemctl: stderr Job for ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038.service failed because a timeout was exceeded. 2023-11-15T04:16:50.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: systemctl: stderr See "systemctl status ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038.service" and "journalctl -xe" for details. 2023-11-15T04:16:50.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: Traceback (most recent call last): 2023-11-15T04:16:50.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T04:16:50.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: main() 2023-11-15T04:16:50.599 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T04:16:50.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: r = ctx.func(ctx) 2023-11-15T04:16:50.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T04:16:50.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: return func(ctx) 2023-11-15T04:16:50.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T04:16:50.600 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: ports=daemon_ports) 2023-11-15T04:16:50.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T04:16:50.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T04:16:50.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T04:16:50.601 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T04:16:50.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: File "/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T04:16:50.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T04:16:50.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: RuntimeError: Failed command: systemctl start ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@node-exporter.smithi038 2023-11-15T04:16:50.602 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:50 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cephadm 2023-11-15T04:16:48.891415+0000 mgr.smithi038.bnbogz (mgr.14164) 247 : cephadm [INF] Deploying daemon prometheus.smithi038 on smithi038 2023-11-15T04:16:50.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.776+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efeb4107020 con 0x7efeb40f69e0 2023-11-15T04:16:50.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.779+0000 7efe9b7fe700 1 -- 172.21.15.188:0/1383209973 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efea8010230 con 0x7efeb40f69e0 2023-11-15T04:16:50.779 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:50.780 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:50.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.779+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 msgr2=0x7efe9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:50.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.779+0000 7efeb9a54700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 0x7efe9c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).stop 2023-11-15T04:16:50.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.779+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 msgr2=0x7efeb41055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:50.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.779+0000 7efeb9a54700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7efea8013510 tx=0x7efea800b040).stop 2023-11-15T04:16:50.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 shutdown_connections 2023-11-15T04:16:50.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7efe9c02ff60 0x7efe9c032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:50.781 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 --2- 172.21.15.188:0/1383209973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb40f69e0 0x7efeb41055f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:50.782 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 >> 172.21.15.188:0/1383209973 conn(0x7efeb40f02e0 msgr2=0x7efeb40f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:50.782 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 shutdown_connections 2023-11-15T04:16:50.782 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:50.780+0000 7efeb9a54700 1 -- 172.21.15.188:0/1383209973 wait complete. 2023-11-15T04:16:50.783 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:50.123312+0000 mgr.smithi038.bnbogz (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:50.215170+0000 mon.smithi038 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:16:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:50.216612+0000 mon.smithi038 (mon.0) 299 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:16:51.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:50.779262+0000 mon.smithi038 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.188:0/1383209973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:52.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:52.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:52.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:52.123622+0000 mgr.smithi038.bnbogz (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:53.573 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- 172.21.15.188:0/3589789112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 msgr2=0x7f50a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:53.573 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 --2- 172.21.15.188:0/3589789112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a80f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5090009fb0 tx=0x7f5090009b30).stop 2023-11-15T04:16:53.573 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- 172.21.15.188:0/3589789112 shutdown_connections 2023-11-15T04:16:53.573 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 --2- 172.21.15.188:0/3589789112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a80f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:53.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- 172.21.15.188:0/3589789112 >> 172.21.15.188:0/3589789112 conn(0x7f50a80f0b70 msgr2=0x7f50a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:53.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- 172.21.15.188:0/3589789112 shutdown_connections 2023-11-15T04:16:53.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- 172.21.15.188:0/3589789112 wait complete. 2023-11-15T04:16:53.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 Processor -- start 2023-11-15T04:16:53.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.572+0000 7f50ade9a700 1 -- start start 2023-11-15T04:16:53.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50ade9a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:53.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50ade9a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a8060240 con 0x7f50a80f5410 2023-11-15T04:16:53.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50a77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:53.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50a77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:44168/0 (socket says 172.21.15.188:44168) 2023-11-15T04:16:53.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50a77fe700 1 -- 172.21.15.188:0/683256831 learned_addr learned my addr 172.21.15.188:0/683256831 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:53.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50a77fe700 1 -- 172.21.15.188:0/683256831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090005040 con 0x7f50a80f5410 2023-11-15T04:16:53.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50a77fe700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5090014000 tx=0x7f509000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:53.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50900095f0 con 0x7f50a80f5410 2023-11-15T04:16:53.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.573+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a8060440 con 0x7f50a80f5410 2023-11-15T04:16:53.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5090007e70 con 0x7f50a80f5410 2023-11-15T04:16:53.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5090014ea0 con 0x7f50a80f5410 2023-11-15T04:16:53.578 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a8109ae0 con 0x7f50a80f5410 2023-11-15T04:16:53.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5090025070 con 0x7f50a80f5410 2023-11-15T04:16:53.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a8045ad0 con 0x7f50a80f5410 2023-11-15T04:16:53.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.574+0000 7f509ffff700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 0x7f5094032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:53.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.575+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5090040970 con 0x7f50a80f5410 2023-11-15T04:16:53.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.575+0000 7f50a6ffd700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 0x7f5094032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:53.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.575+0000 7f50a6ffd700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 0x7f5094032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f50980096a0 tx=0x7f5098006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:53.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.580+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50900079c0 con 0x7f50a80f5410 2023-11-15T04:16:53.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.783+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50a80fae60 con 0x7f50a80f5410 2023-11-15T04:16:53.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.784+0000 7f509ffff700 1 -- 172.21.15.188:0/683256831 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f509000f300 con 0x7f50a80f5410 2023-11-15T04:16:53.785 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:53.785 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:53.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 msgr2=0x7f5094032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:53.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 0x7f5094032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f50980096a0 tx=0x7f5098006b40).stop 2023-11-15T04:16:53.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 msgr2=0x7f50a8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:53.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5090014000 tx=0x7f509000b040).stop 2023-11-15T04:16:53.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 shutdown_connections 2023-11-15T04:16:53.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f509402ff60 0x7f5094032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:53.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 --2- 172.21.15.188:0/683256831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50a80f5410 0x7f50a8109430 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:53.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 >> 172.21.15.188:0/683256831 conn(0x7f50a80f0b70 msgr2=0x7f50a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:53.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 shutdown_connections 2023-11-15T04:16:53.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:53.785+0000 7f50ade9a700 1 -- 172.21.15.188:0/683256831 wait complete. 2023-11-15T04:16:53.788 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:54.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:53.785049+0000 mon.smithi038 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.188:0/683256831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:55.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:54 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:54.124019+0000 mgr.smithi038.bnbogz (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:55.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:55.220 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:56.418 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- 172.21.15.188:0/598659723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 msgr2=0x7ff75c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:56.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 --2- 172.21.15.188:0/598659723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c0f68a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7ff74c009fb0 tx=0x7ff74c009b30).stop 2023-11-15T04:16:56.419 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- 172.21.15.188:0/598659723 shutdown_connections 2023-11-15T04:16:56.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 --2- 172.21.15.188:0/598659723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c0f68a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:56.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- 172.21.15.188:0/598659723 >> 172.21.15.188:0/598659723 conn(0x7ff75c0f1c00 msgr2=0x7ff75c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:56.420 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- 172.21.15.188:0/598659723 shutdown_connections 2023-11-15T04:16:56.421 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- 172.21.15.188:0/598659723 wait complete. 2023-11-15T04:16:56.423 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 Processor -- start 2023-11-15T04:16:56.423 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.418+0000 7ff763e5f700 1 -- start start 2023-11-15T04:16:56.423 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff763e5f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:56.424 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff763e5f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c106bc0 con 0x7ff75c0f6480 2023-11-15T04:16:56.424 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff761bfb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:56.424 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff761bfb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:44186/0 (socket says 172.21.15.188:44186) 2023-11-15T04:16:56.424 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff761bfb700 1 -- 172.21.15.188:0/2848746844 learned_addr learned my addr 172.21.15.188:0/2848746844 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:56.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff761bfb700 1 -- 172.21.15.188:0/2848746844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c005040 con 0x7ff75c0f6480 2023-11-15T04:16:56.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.419+0000 7ff761bfb700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ff74c013510 tx=0x7ff74c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:56.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.420+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c009320 con 0x7ff75c0f6480 2023-11-15T04:16:56.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.420+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff74c013f00 con 0x7ff75c0f6480 2023-11-15T04:16:56.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.420+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c014d90 con 0x7ff75c0f6480 2023-11-15T04:16:56.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.420+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c106dc0 con 0x7ff75c0f6480 2023-11-15T04:16:56.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.420+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c107250 con 0x7ff75c0f6480 2023-11-15T04:16:56.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.421+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c045ad0 con 0x7ff75c0f6480 2023-11-15T04:16:56.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.421+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff74c0095f0 con 0x7ff75c0f6480 2023-11-15T04:16:56.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.421+0000 7ff7527fc700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 0x7ff748032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:56.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.421+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff74c041690 con 0x7ff75c0f6480 2023-11-15T04:16:56.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.421+0000 7ff7613fa700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 0x7ff748032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:56.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.422+0000 7ff7613fa700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 0x7ff748032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff7580096a0 tx=0x7ff758006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:56.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.426+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74c014420 con 0x7ff75c0f6480 2023-11-15T04:16:56.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.630+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff75c0fbf40 con 0x7ff75c0f6480 2023-11-15T04:16:56.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.631+0000 7ff7527fc700 1 -- 172.21.15.188:0/2848746844 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff74c02d090 con 0x7ff75c0f6480 2023-11-15T04:16:56.632 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:56.632 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:56.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 msgr2=0x7ff748032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 0x7ff748032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff7580096a0 tx=0x7ff758006b40).stop 2023-11-15T04:16:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 msgr2=0x7ff75c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ff74c013510 tx=0x7ff74c00b040).stop 2023-11-15T04:16:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 shutdown_connections 2023-11-15T04:16:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7ff74802ff60 0x7ff748032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:56.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 --2- 172.21.15.188:0/2848746844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c0f6480 0x7ff75c1066d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:56.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 >> 172.21.15.188:0/2848746844 conn(0x7ff75c0f1c00 msgr2=0x7ff75c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:56.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 shutdown_connections 2023-11-15T04:16:56.635 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:56.633+0000 7ff763e5f700 1 -- 172.21.15.188:0/2848746844 wait complete. 2023-11-15T04:16:56.636 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:56.124369+0000 mgr.smithi038.bnbogz (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:16:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:56.632352+0000 mon.smithi038 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.188:0/2848746844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:16:58.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:16:58.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:16:59.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 -- 172.21.15.188:0/3592288185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 msgr2=0x7f77b40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:59.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 --2- 172.21.15.188:0/3592288185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b40f66c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f77a4009fb0 tx=0x7f77a4009b30).stop 2023-11-15T04:16:59.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 -- 172.21.15.188:0/3592288185 shutdown_connections 2023-11-15T04:16:59.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 --2- 172.21.15.188:0/3592288185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b40f66c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:59.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 -- 172.21.15.188:0/3592288185 >> 172.21.15.188:0/3592288185 conn(0x7f77b40f1aa0 msgr2=0x7f77b40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:59.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 -- 172.21.15.188:0/3592288185 shutdown_connections 2023-11-15T04:16:59.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.217+0000 7f77b3fff700 1 -- 172.21.15.188:0/3592288185 wait complete. 2023-11-15T04:16:59.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b3fff700 1 Processor -- start 2023-11-15T04:16:59.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b3fff700 1 -- start start 2023-11-15T04:16:59.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:59.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b3fff700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b41069a0 con 0x7f77b40f62a0 2023-11-15T04:16:59.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:59.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36098/0 (socket says 172.21.15.188:36098) 2023-11-15T04:16:59.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.218+0000 7f77b2ffd700 1 -- 172.21.15.188:0/381221656 learned_addr learned my addr 172.21.15.188:0/381221656 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:16:59.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f77b2ffd700 1 -- 172.21.15.188:0/381221656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a4005040 con 0x7f77b40f62a0 2023-11-15T04:16:59.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f77b2ffd700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f77a4013510 tx=0x7f77a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:59.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a4009320 con 0x7f77b40f62a0 2023-11-15T04:16:59.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77a4013f00 con 0x7f77b40f62a0 2023-11-15T04:16:59.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b4106ba0 con 0x7f77b40f62a0 2023-11-15T04:16:59.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a4014dc0 con 0x7f77b40f62a0 2023-11-15T04:16:59.223 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.219+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b4107030 con 0x7f77b40f62a0 2023-11-15T04:16:59.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.220+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77a4014420 con 0x7f77b40f62a0 2023-11-15T04:16:59.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.220+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b4045ad0 con 0x7f77b40f62a0 2023-11-15T04:16:59.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.220+0000 7f779b7fe700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:16:59.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.220+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77a4041110 con 0x7f77b40f62a0 2023-11-15T04:16:59.224 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.221+0000 7f77b27fc700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:16:59.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.222+0000 7f77b27fc700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 0x7f779c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f77a80096a0 tx=0x7f77a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:16:59.225 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.225+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77a40095f0 con 0x7f77b40f62a0 2023-11-15T04:16:59.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.428+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77b4107ee0 con 0x7f77b40f62a0 2023-11-15T04:16:59.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.429+0000 7f779b7fe700 1 -- 172.21.15.188:0/381221656 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77a401cb10 con 0x7f77b40f62a0 2023-11-15T04:16:59.429 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:16:59.430 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:16:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 msgr2=0x7f779c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 0x7f779c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f77a80096a0 tx=0x7f77a8006b40).stop 2023-11-15T04:16:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 msgr2=0x7f77b41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:16:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f77a4013510 tx=0x7f77a400b040).stop 2023-11-15T04:16:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 shutdown_connections 2023-11-15T04:16:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 --2- 172.21.15.188:0/381221656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77b40f62a0 0x7f77b41064b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:16:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 >> 172.21.15.188:0/381221656 conn(0x7f77b40f1aa0 msgr2=0x7f77b40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:16:59.433 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 shutdown_connections 2023-11-15T04:16:59.433 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:16:59.431+0000 7f77b3fff700 1 -- 172.21.15.188:0/381221656 wait complete. 2023-11-15T04:16:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:16:59.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:16:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:16:58.124777+0000 mgr.smithi038.bnbogz (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:00.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:00 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:16:59.429517+0000 mon.smithi038 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.188:0/381221656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:00.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:00.831 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:00.125206+0000 mgr.smithi038.bnbogz (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:02.136 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 -- 172.21.15.188:0/167364656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 msgr2=0x7f3af80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 --2- 172.21.15.188:0/167364656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af80f66c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f3ae8009fb0 tx=0x7f3ae8009b30).stop 2023-11-15T04:17:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 -- 172.21.15.188:0/167364656 shutdown_connections 2023-11-15T04:17:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 --2- 172.21.15.188:0/167364656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af80f66c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 -- 172.21.15.188:0/167364656 >> 172.21.15.188:0/167364656 conn(0x7f3af80f1aa0 msgr2=0x7f3af80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 -- 172.21.15.188:0/167364656 shutdown_connections 2023-11-15T04:17:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.135+0000 7f3af759e700 1 -- 172.21.15.188:0/167364656 wait complete. 2023-11-15T04:17:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af759e700 1 Processor -- start 2023-11-15T04:17:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af759e700 1 -- start start 2023-11-15T04:17:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af759e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af759e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af81069a0 con 0x7f3af80f62a0 2023-11-15T04:17:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36118/0 (socket says 172.21.15.188:36118) 2023-11-15T04:17:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af659c700 1 -- 172.21.15.188:0/4290826343 learned_addr learned my addr 172.21.15.188:0/4290826343 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.136+0000 7f3af659c700 1 -- 172.21.15.188:0/4290826343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8005040 con 0x7f3af80f62a0 2023-11-15T04:17:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3af659c700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:02.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8009320 con 0x7f3af80f62a0 2023-11-15T04:17:02.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af8106ba0 con 0x7f3af80f62a0 2023-11-15T04:17:02.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae8013f00 con 0x7f3af80f62a0 2023-11-15T04:17:02.140 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8014dc0 con 0x7f3af80f62a0 2023-11-15T04:17:02.141 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.137+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af8107030 con 0x7f3af80f62a0 2023-11-15T04:17:02.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae80095f0 con 0x7f3af80f62a0 2023-11-15T04:17:02.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af8045ad0 con 0x7f3af80f62a0 2023-11-15T04:17:02.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3ae6ffd700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 0x7f3ae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:02.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae8041040 con 0x7f3af80f62a0 2023-11-15T04:17:02.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3af5d9b700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 0x7f3ae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:02.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.138+0000 7f3af5d9b700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 0x7f3ae0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3aec0096a0 tx=0x7f3aec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:02.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.143+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae8014420 con 0x7f3af80f62a0 2023-11-15T04:17:02.347 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.347+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af8107ee0 con 0x7f3af80f62a0 2023-11-15T04:17:02.348 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.348+0000 7f3ae6ffd700 1 -- 172.21.15.188:0/4290826343 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ae8010230 con 0x7f3af80f62a0 2023-11-15T04:17:02.349 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:02.349 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 msgr2=0x7f3ae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 0x7f3ae0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3aec0096a0 tx=0x7f3aec006b40).stop 2023-11-15T04:17:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 msgr2=0x7f3af81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:02.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).stop 2023-11-15T04:17:02.352 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 shutdown_connections 2023-11-15T04:17:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f3ae002ff60 0x7f3ae0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 --2- 172.21.15.188:0/4290826343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3af80f62a0 0x7f3af81064b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 >> 172.21.15.188:0/4290826343 conn(0x7f3af80f1aa0 msgr2=0x7f3af80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:02.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 shutdown_connections 2023-11-15T04:17:02.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:02.350+0000 7f3af759e700 1 -- 172.21.15.188:0/4290826343 wait complete. 2023-11-15T04:17:02.354 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:02.125538+0000 mgr.smithi038.bnbogz (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:02.348444+0000 mon.smithi038 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.188:0/4290826343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:03.757 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:03.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:05.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- 172.21.15.188:0/1550763299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 msgr2=0x7f02b00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 --2- 172.21.15.188:0/1550763299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b00f8f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0298009fb0 tx=0x7f0298009b30).stop 2023-11-15T04:17:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- 172.21.15.188:0/1550763299 shutdown_connections 2023-11-15T04:17:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 --2- 172.21.15.188:0/1550763299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b00f8f20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- 172.21.15.188:0/1550763299 >> 172.21.15.188:0/1550763299 conn(0x7f02b00f09a0 msgr2=0x7f02b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- 172.21.15.188:0/1550763299 shutdown_connections 2023-11-15T04:17:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- 172.21.15.188:0/1550763299 wait complete. 2023-11-15T04:17:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 Processor -- start 2023-11-15T04:17:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.021+0000 7f02b5606700 1 -- start start 2023-11-15T04:17:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02b5606700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02b5606700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01069f0 con 0x7f02b00601e0 2023-11-15T04:17:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36126/0 (socket says 172.21.15.188:36126) 2023-11-15T04:17:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02affff700 1 -- 172.21.15.188:0/1455815692 learned_addr learned my addr 172.21.15.188:0/1455815692 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02affff700 1 -- 172.21.15.188:0/1455815692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298005040 con 0x7f02b00601e0 2023-11-15T04:17:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.022+0000 7f02affff700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0298013510 tx=0x7f029800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.023+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0298009320 con 0x7f02b00601e0 2023-11-15T04:17:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.023+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0106bf0 con 0x7f02b00601e0 2023-11-15T04:17:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.023+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0298013f00 con 0x7f02b00601e0 2023-11-15T04:17:05.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.023+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0298014d90 con 0x7f02b00601e0 2023-11-15T04:17:05.027 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.023+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0107080 con 0x7f02b00601e0 2023-11-15T04:17:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.024+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0045ad0 con 0x7f02b00601e0 2023-11-15T04:17:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.024+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02980095f0 con 0x7f02b00601e0 2023-11-15T04:17:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.024+0000 7f02acff9700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 0x7f029c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.024+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0298041000 con 0x7f02b00601e0 2023-11-15T04:17:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.024+0000 7f02af7fe700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 0x7f029c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:05.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.025+0000 7f02af7fe700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 0x7f029c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f02a00096a0 tx=0x7f02a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:05.029 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.029+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0298014420 con 0x7f02b00601e0 2023-11-15T04:17:05.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.233+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02b0107f30 con 0x7f02b00601e0 2023-11-15T04:17:05.233 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.233+0000 7f02acff9700 1 -- 172.21.15.188:0/1455815692 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0298010230 con 0x7f02b00601e0 2023-11-15T04:17:05.233 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:05.234 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 msgr2=0x7f029c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 0x7f029c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f02a00096a0 tx=0x7f02a0006b40).stop 2023-11-15T04:17:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 msgr2=0x7f02b0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0298013510 tx=0x7f029800b040).stop 2023-11-15T04:17:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 shutdown_connections 2023-11-15T04:17:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f029c02ff60 0x7f029c032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 --2- 172.21.15.188:0/1455815692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b00601e0 0x7f02b0106500 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 >> 172.21.15.188:0/1455815692 conn(0x7f02b00f09a0 msgr2=0x7f02b00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 shutdown_connections 2023-11-15T04:17:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:05.235+0000 7f02b5606700 1 -- 172.21.15.188:0/1455815692 wait complete. 2023-11-15T04:17:05.237 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:05.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:04.125874+0000 mgr.smithi038.bnbogz (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:06.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:06 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:05.233871+0000 mon.smithi038 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.188:0/1455815692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:06.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:06.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:07.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:07 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:06.126243+0000 mgr.smithi038.bnbogz (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:08.003 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 -- 172.21.15.188:0/422105488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 msgr2=0x7f25f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:08.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 --2- 172.21.15.188:0/422105488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f80f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f25e8009fb0 tx=0x7f25e8009b30).stop 2023-11-15T04:17:08.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 -- 172.21.15.188:0/422105488 shutdown_connections 2023-11-15T04:17:08.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 --2- 172.21.15.188:0/422105488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f80f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:08.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 -- 172.21.15.188:0/422105488 >> 172.21.15.188:0/422105488 conn(0x7f25f80f0b70 msgr2=0x7f25f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:08.004 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 -- 172.21.15.188:0/422105488 shutdown_connections 2023-11-15T04:17:08.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 -- 172.21.15.188:0/422105488 wait complete. 2023-11-15T04:17:08.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.002+0000 7f25fca8a700 1 Processor -- start 2023-11-15T04:17:08.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25fca8a700 1 -- start start 2023-11-15T04:17:08.005 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25fca8a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:08.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25fca8a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8106c10 con 0x7f25f80f5410 2023-11-15T04:17:08.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25f659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:08.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25f659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:36148/0 (socket says 172.21.15.188:36148) 2023-11-15T04:17:08.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25f659c700 1 -- 172.21.15.188:0/3087356817 learned_addr learned my addr 172.21.15.188:0/3087356817 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:08.006 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25f659c700 1 -- 172.21.15.188:0/3087356817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e8005040 con 0x7f25f80f5410 2023-11-15T04:17:08.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25f659c700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f25e8013510 tx=0x7f25e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:08.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.003+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25e8009320 con 0x7f25f80f5410 2023-11-15T04:17:08.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f8106e10 con 0x7f25f80f5410 2023-11-15T04:17:08.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25e8013f00 con 0x7f25f80f5410 2023-11-15T04:17:08.007 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25e8014d90 con 0x7f25f80f5410 2023-11-15T04:17:08.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f81072a0 con 0x7f25f80f5410 2023-11-15T04:17:08.008 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f8045ad0 con 0x7f25f80f5410 2023-11-15T04:17:08.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25e80095f0 con 0x7f25f80f5410 2023-11-15T04:17:08.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.004+0000 7f25e6ffd700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 0x7f25e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:08.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.005+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25e8040fe0 con 0x7f25f80f5410 2023-11-15T04:17:08.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.005+0000 7f25f5d9b700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 0x7f25e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:08.009 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.005+0000 7f25f5d9b700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 0x7f25e0032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f25ec0096a0 tx=0x7f25ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:08.010 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.009+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25e806d2b0 con 0x7f25f80f5410 2023-11-15T04:17:08.213 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.214+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25f8108150 con 0x7f25f80f5410 2023-11-15T04:17:08.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.214+0000 7f25e6ffd700 1 -- 172.21.15.188:0/3087356817 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25e8014420 con 0x7f25f80f5410 2023-11-15T04:17:08.214 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:08.215 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:08.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 msgr2=0x7f25e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:08.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 0x7f25e0032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f25ec0096a0 tx=0x7f25ec006b40).stop 2023-11-15T04:17:08.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 msgr2=0x7f25f8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f25e8013510 tx=0x7f25e800b040).stop 2023-11-15T04:17:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 shutdown_connections 2023-11-15T04:17:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f25e002ffb0 0x7f25e0032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 --2- 172.21.15.188:0/3087356817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25f80f5410 0x7f25f8106720 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:08.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 >> 172.21.15.188:0/3087356817 conn(0x7f25f80f0b70 msgr2=0x7f25f805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:08.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 shutdown_connections 2023-11-15T04:17:08.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:08.216+0000 7f25fca8a700 1 -- 172.21.15.188:0/3087356817 wait complete. 2023-11-15T04:17:08.218 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:08.126614+0000 mgr.smithi038.bnbogz (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:09.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:09 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:08.214814+0000 mon.smithi038 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.188:0/3087356817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:09.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:09.636 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:10.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 -- 172.21.15.188:0/702139487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 msgr2=0x7fb8240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:10.826 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 --2- 172.21.15.188:0/702139487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb8240f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fb818009fb0 tx=0x7fb818009b30).stop 2023-11-15T04:17:10.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 -- 172.21.15.188:0/702139487 shutdown_connections 2023-11-15T04:17:10.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 --2- 172.21.15.188:0/702139487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb8240f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:10.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 -- 172.21.15.188:0/702139487 >> 172.21.15.188:0/702139487 conn(0x7fb8240f0b70 msgr2=0x7fb8240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:10.827 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 -- 172.21.15.188:0/702139487 shutdown_connections 2023-11-15T04:17:10.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.823+0000 7fb82b177700 1 -- 172.21.15.188:0/702139487 wait complete. 2023-11-15T04:17:10.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb82b177700 1 Processor -- start 2023-11-15T04:17:10.828 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb82b177700 1 -- start start 2023-11-15T04:17:10.829 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb82b177700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:10.829 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb82b177700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824060240 con 0x7fb8240f5410 2023-11-15T04:17:10.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb828f13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:10.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb828f13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51834/0 (socket says 172.21.15.188:51834) 2023-11-15T04:17:10.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.824+0000 7fb828f13700 1 -- 172.21.15.188:0/3221316621 learned_addr learned my addr 172.21.15.188:0/3221316621 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:10.830 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb828f13700 1 -- 172.21.15.188:0/3221316621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818005040 con 0x7fb8240f5410 2023-11-15T04:17:10.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb828f13700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb818013510 tx=0x7fb81800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:10.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb818009320 con 0x7fb8240f5410 2023-11-15T04:17:10.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb818013f00 con 0x7fb8240f5410 2023-11-15T04:17:10.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824060440 con 0x7fb8240f5410 2023-11-15T04:17:10.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb818014d90 con 0x7fb8240f5410 2023-11-15T04:17:10.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.825+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb824109ae0 con 0x7fb8240f5410 2023-11-15T04:17:10.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.826+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8180263a0 con 0x7fb8240f5410 2023-11-15T04:17:10.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.826+0000 7fb8217fa700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 0x7fb814032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:10.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.826+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb818041380 con 0x7fb8240f5410 2023-11-15T04:17:10.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.826+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824045ad0 con 0x7fb8240f5410 2023-11-15T04:17:10.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.826+0000 7fb823fff700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 0x7fb814032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:10.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.827+0000 7fb823fff700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 0x7fb814032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:10.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:10.831+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb818012920 con 0x7fb8240f5410 2023-11-15T04:17:11.036 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.036+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb82410a230 con 0x7fb8240f5410 2023-11-15T04:17:11.037 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.036+0000 7fb8217fa700 1 -- 172.21.15.188:0/3221316621 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb81802b030 con 0x7fb8240f5410 2023-11-15T04:17:11.037 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:11.038 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:11.038 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 msgr2=0x7fb814032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:11.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 0x7fb814032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).stop 2023-11-15T04:17:11.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 msgr2=0x7fb824109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:11.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb818013510 tx=0x7fb81800b040).stop 2023-11-15T04:17:11.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 shutdown_connections 2023-11-15T04:17:11.039 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb81402ff60 0x7fb814032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:11.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 --2- 172.21.15.188:0/3221316621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8240f5410 0x7fb824109430 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:11.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 >> 172.21.15.188:0/3221316621 conn(0x7fb8240f0b70 msgr2=0x7fb8240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:11.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 shutdown_connections 2023-11-15T04:17:11.040 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:11.038+0000 7fb82b177700 1 -- 172.21.15.188:0/3221316621 wait complete. 2023-11-15T04:17:11.041 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:11.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:10.127045+0000 mgr.smithi038.bnbogz (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:11.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:11 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:11.037750+0000 mon.smithi038 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.188:0/3221316621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:12.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:12.432 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:12.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:12 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:12.127413+0000 mgr.smithi038.bnbogz (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:13.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 -- 172.21.15.188:0/773550707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 msgr2=0x7f05a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:13.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 --2- 172.21.15.188:0/773550707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a80f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f059c009fb0 tx=0x7f059c009b30).stop 2023-11-15T04:17:13.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 -- 172.21.15.188:0/773550707 shutdown_connections 2023-11-15T04:17:13.674 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 --2- 172.21.15.188:0/773550707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a80f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:13.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 -- 172.21.15.188:0/773550707 >> 172.21.15.188:0/773550707 conn(0x7f05a80f0b70 msgr2=0x7f05a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:13.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 -- 172.21.15.188:0/773550707 shutdown_connections 2023-11-15T04:17:13.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.672+0000 7f05aec2d700 1 -- 172.21.15.188:0/773550707 wait complete. 2023-11-15T04:17:13.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.673+0000 7f05aec2d700 1 Processor -- start 2023-11-15T04:17:13.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.673+0000 7f05aec2d700 1 -- start start 2023-11-15T04:17:13.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.673+0000 7f05aec2d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:13.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.673+0000 7f05aec2d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8105b20 con 0x7f05a80f5410 2023-11-15T04:17:13.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.673+0000 7f05ac9c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:13.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05ac9c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51846/0 (socket says 172.21.15.188:51846) 2023-11-15T04:17:13.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05ac9c9700 1 -- 172.21.15.188:0/899589779 learned_addr learned my addr 172.21.15.188:0/899589779 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:13.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05ac9c9700 1 -- 172.21.15.188:0/899589779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c005040 con 0x7f05a80f5410 2023-11-15T04:17:13.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05ac9c9700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f059c014000 tx=0x7f059c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:13.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c0095f0 con 0x7f05a80f5410 2023-11-15T04:17:13.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a8105d20 con 0x7f05a80f5410 2023-11-15T04:17:13.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f059c007e70 con 0x7f05a80f5410 2023-11-15T04:17:13.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059c014ea0 con 0x7f05a80f5410 2023-11-15T04:17:13.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.674+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a81061b0 con 0x7f05a80f5410 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.675+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f059c027070 con 0x7f05a80f5410 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.675+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a8045ad0 con 0x7f05a80f5410 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.675+0000 7f05a57fa700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 0x7f0598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.675+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f059c042710 con 0x7f05a80f5410 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.676+0000 7f05a7fff700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 0x7f0598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:13.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.676+0000 7f05a7fff700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 0x7f0598032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f05940096a0 tx=0x7f0594006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:13.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.681+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f059c0274e0 con 0x7f05a80f5410 2023-11-15T04:17:13.886 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.886+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05a80fae60 con 0x7f05a80f5410 2023-11-15T04:17:13.887 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.886+0000 7f05a57fa700 1 -- 172.21.15.188:0/899589779 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f059c027300 con 0x7f05a80f5410 2023-11-15T04:17:13.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:13.888 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:13.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 msgr2=0x7f0598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:13.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 0x7f0598032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f05940096a0 tx=0x7f0594006b40).stop 2023-11-15T04:17:13.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 msgr2=0x7f05a8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:13.889 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f059c014000 tx=0x7f059c00b040).stop 2023-11-15T04:17:13.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 shutdown_connections 2023-11-15T04:17:13.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f059802ff60 0x7f0598032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:13.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 --2- 172.21.15.188:0/899589779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a80f5410 0x7f05a8105630 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:13.890 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 >> 172.21.15.188:0/899589779 conn(0x7f05a80f0b70 msgr2=0x7f05a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 shutdown_connections 2023-11-15T04:17:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:13.888+0000 7f05aec2d700 1 -- 172.21.15.188:0/899589779 wait complete. 2023-11-15T04:17:13.891 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:14.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:13 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:13.887839+0000 mon.smithi038 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.188:0/899589779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:15.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:15.282 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:15.342 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:14 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:14.127790+0000 mgr.smithi038.bnbogz (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:16.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 -- 172.21.15.188:0/2937623749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 msgr2=0x7f9d300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:16.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 --2- 172.21.15.188:0/2937623749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d300f6930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9d24009fb0 tx=0x7f9d24009b30).stop 2023-11-15T04:17:16.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 -- 172.21.15.188:0/2937623749 shutdown_connections 2023-11-15T04:17:16.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 --2- 172.21.15.188:0/2937623749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d300f6930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:16.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 -- 172.21.15.188:0/2937623749 >> 172.21.15.188:0/2937623749 conn(0x7f9d300f1c70 msgr2=0x7f9d300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:16.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 -- 172.21.15.188:0/2937623749 shutdown_connections 2023-11-15T04:17:16.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.512+0000 7f9d355a2700 1 -- 172.21.15.188:0/2937623749 wait complete. 2023-11-15T04:17:16.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d355a2700 1 Processor -- start 2023-11-15T04:17:16.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d355a2700 1 -- start start 2023-11-15T04:17:16.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d355a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:16.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d355a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30106c20 con 0x7f9d300f6510 2023-11-15T04:17:16.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:16.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d2effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:51872/0 (socket says 172.21.15.188:51872) 2023-11-15T04:17:16.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d2effd700 1 -- 172.21.15.188:0/1530743302 learned_addr learned my addr 172.21.15.188:0/1530743302 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:16.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d2effd700 1 -- 172.21.15.188:0/1530743302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24005040 con 0x7f9d300f6510 2023-11-15T04:17:16.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.513+0000 7f9d2effd700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9d24014000 tx=0x7f9d2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:16.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d240095f0 con 0x7f9d300f6510 2023-11-15T04:17:16.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d24007e70 con 0x7f9d300f6510 2023-11-15T04:17:16.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d30106e20 con 0x7f9d300f6510 2023-11-15T04:17:16.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d24014ea0 con 0x7f9d300f6510 2023-11-15T04:17:16.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d301072b0 con 0x7f9d300f6510 2023-11-15T04:17:16.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.514+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d24025070 con 0x7f9d300f6510 2023-11-15T04:17:16.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.515+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d30045ad0 con 0x7f9d300f6510 2023-11-15T04:17:16.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.515+0000 7f9d177fe700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 0x7f9d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:16.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.515+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d24040a30 con 0x7f9d300f6510 2023-11-15T04:17:16.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.515+0000 7f9d2e7fc700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 0x7f9d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:16.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.515+0000 7f9d2e7fc700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 0x7f9d18032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9d200096a0 tx=0x7f9d20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:16.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.520+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d240079c0 con 0x7f9d300f6510 2023-11-15T04:17:16.728 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.727+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d300fbf60 con 0x7f9d300f6510 2023-11-15T04:17:16.728 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.728+0000 7f9d177fe700 1 -- 172.21.15.188:0/1530743302 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d2400f300 con 0x7f9d300f6510 2023-11-15T04:17:16.729 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:16.729 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:16.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 msgr2=0x7f9d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:16.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 0x7f9d18032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9d200096a0 tx=0x7f9d20006b40).stop 2023-11-15T04:17:16.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 msgr2=0x7f9d30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:16.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9d24014000 tx=0x7f9d2400b040).stop 2023-11-15T04:17:16.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 shutdown_connections 2023-11-15T04:17:16.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f9d1802ff60 0x7f9d18032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:16.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 --2- 172.21.15.188:0/1530743302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d300f6510 0x7f9d30106730 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:16.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 >> 172.21.15.188:0/1530743302 conn(0x7f9d300f1c70 msgr2=0x7f9d300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:16.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 shutdown_connections 2023-11-15T04:17:16.732 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:16.729+0000 7f9d355a2700 1 -- 172.21.15.188:0/1530743302 wait complete. 2023-11-15T04:17:16.732 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:16.128097+0000 mgr.smithi038.bnbogz (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:17.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:17 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:16.729007+0000 mon.smithi038 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.188:0/1530743302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:18.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:18.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:19.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- 172.21.15.188:0/427145394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 msgr2=0x7f0a800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:19.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 --2- 172.21.15.188:0/427145394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a800f90f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f0a70009fb0 tx=0x7f0a70009b30).stop 2023-11-15T04:17:19.379 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- 172.21.15.188:0/427145394 shutdown_connections 2023-11-15T04:17:19.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 --2- 172.21.15.188:0/427145394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a800f90f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:19.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- 172.21.15.188:0/427145394 >> 172.21.15.188:0/427145394 conn(0x7f0a800f0b70 msgr2=0x7f0a800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:19.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- 172.21.15.188:0/427145394 shutdown_connections 2023-11-15T04:17:19.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- 172.21.15.188:0/427145394 wait complete. 2023-11-15T04:17:19.380 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 Processor -- start 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.377+0000 7f0a85da3700 1 -- start start 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a85da3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a85da3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a80106bd0 con 0x7f0a80060280 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a7f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a7f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56320/0 (socket says 172.21.15.188:56320) 2023-11-15T04:17:19.381 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a7f7fe700 1 -- 172.21.15.188:0/3176171057 learned_addr learned my addr 172.21.15.188:0/3176171057 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:19.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a7f7fe700 1 -- 172.21.15.188:0/3176171057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a70005040 con 0x7f0a80060280 2023-11-15T04:17:19.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a7f7fe700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0a70013510 tx=0x7f0a7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:19.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a70009320 con 0x7f0a80060280 2023-11-15T04:17:19.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a80106dd0 con 0x7f0a80060280 2023-11-15T04:17:19.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.378+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a70013f00 con 0x7f0a80060280 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.379+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a70014dc0 con 0x7f0a80060280 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.379+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a80107260 con 0x7f0a80060280 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.379+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a700095f0 con 0x7f0a80060280 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.379+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a80045ad0 con 0x7f0a80060280 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.379+0000 7f0a77fff700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 0x7f0a6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:19.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.380+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a70041020 con 0x7f0a80060280 2023-11-15T04:17:19.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.380+0000 7f0a7effd700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 0x7f0a6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:19.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.380+0000 7f0a7effd700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 0x7f0a6c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:19.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.384+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a70014420 con 0x7f0a80060280 2023-11-15T04:17:19.590 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.589+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a801080b0 con 0x7f0a80060280 2023-11-15T04:17:19.590 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.590+0000 7f0a77fff700 1 -- 172.21.15.188:0/3176171057 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a7001c920 con 0x7f0a80060280 2023-11-15T04:17:19.591 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:19.591 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:19.592 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 msgr2=0x7f0a6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:19.592 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 0x7f0a6c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).stop 2023-11-15T04:17:19.592 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 msgr2=0x7f0a801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:19.592 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0a70013510 tx=0x7f0a7000b040).stop 2023-11-15T04:17:19.593 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 shutdown_connections 2023-11-15T04:17:19.593 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f0a6c02ff60 0x7f0a6c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:19.593 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 --2- 172.21.15.188:0/3176171057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a80060280 0x7f0a801066e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:19.593 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 >> 172.21.15.188:0/3176171057 conn(0x7f0a800f0b70 msgr2=0x7f0a800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:19.593 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 shutdown_connections 2023-11-15T04:17:19.594 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:19.591+0000 7f0a85da3700 1 -- 172.21.15.188:0/3176171057 wait complete. 2023-11-15T04:17:19.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:19 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:18.128493+0000 mgr.smithi038.bnbogz (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:19.595 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:20.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:20 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:19.591105+0000 mon.smithi038 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.188:0/3176171057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:20.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:20.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:21.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:21 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:20.128947+0000 mgr.smithi038.bnbogz (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:22.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 -- 172.21.15.188:0/2410691817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 msgr2=0x7fc0b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:22.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 --2- 172.21.15.188:0/2410691817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b40f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fc0a8009fb0 tx=0x7fc0a8009b30).stop 2023-11-15T04:17:22.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 -- 172.21.15.188:0/2410691817 shutdown_connections 2023-11-15T04:17:22.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 --2- 172.21.15.188:0/2410691817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b40f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:22.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 -- 172.21.15.188:0/2410691817 >> 172.21.15.188:0/2410691817 conn(0x7fc0b40f0b70 msgr2=0x7fc0b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:22.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 -- 172.21.15.188:0/2410691817 shutdown_connections 2023-11-15T04:17:22.310 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.307+0000 7fc0bab91700 1 -- 172.21.15.188:0/2410691817 wait complete. 2023-11-15T04:17:22.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0bab91700 1 Processor -- start 2023-11-15T04:17:22.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0bab91700 1 -- start start 2023-11-15T04:17:22.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0bab91700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:22.311 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0bab91700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4060280 con 0x7fc0b40f5410 2023-11-15T04:17:22.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0b892d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:22.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0b892d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56340/0 (socket says 172.21.15.188:56340) 2023-11-15T04:17:22.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.308+0000 7fc0b892d700 1 -- 172.21.15.188:0/128633257 learned_addr learned my addr 172.21.15.188:0/128633257 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:22.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0b892d700 1 -- 172.21.15.188:0/128633257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a8005040 con 0x7fc0b40f5410 2023-11-15T04:17:22.312 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0b892d700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc0a8013510 tx=0x7fc0a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:22.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a8009320 con 0x7fc0b40f5410 2023-11-15T04:17:22.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b4060480 con 0x7fc0b40f5410 2023-11-15T04:17:22.314 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0a8013f00 con 0x7fc0b40f5410 2023-11-15T04:17:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a8014d90 con 0x7fc0b40f5410 2023-11-15T04:17:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.309+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b4109b20 con 0x7fc0b40f5410 2023-11-15T04:17:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.310+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0a80095f0 con 0x7fc0b40f5410 2023-11-15T04:17:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.310+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b4045ad0 con 0x7fc0b40f5410 2023-11-15T04:17:22.315 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.310+0000 7fc0b17fa700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 0x7fc0a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:22.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.310+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0a8041000 con 0x7fc0b40f5410 2023-11-15T04:17:22.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.310+0000 7fc0b3fff700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 0x7fc0a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:22.316 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.311+0000 7fc0b3fff700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 0x7fc0a4032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc0a00096a0 tx=0x7fc0a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:22.317 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.315+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0a8014420 con 0x7fc0b40f5410 2023-11-15T04:17:22.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.519+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0b410a340 con 0x7fc0b40f5410 2023-11-15T04:17:22.521 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.520+0000 7fc0b17fa700 1 -- 172.21.15.188:0/128633257 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0a801d7f0 con 0x7fc0b40f5410 2023-11-15T04:17:22.521 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:22.521 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:22.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 msgr2=0x7fc0a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:22.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 0x7fc0a4032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc0a00096a0 tx=0x7fc0a0006b40).stop 2023-11-15T04:17:22.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 msgr2=0x7fc0b41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:22.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc0a8013510 tx=0x7fc0a800b040).stop 2023-11-15T04:17:22.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 shutdown_connections 2023-11-15T04:17:22.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc0a402ff60 0x7fc0a4032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:22.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 --2- 172.21.15.188:0/128633257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b40f5410 0x7fc0b41093f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:22.524 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 >> 172.21.15.188:0/128633257 conn(0x7fc0b40f0b70 msgr2=0x7fc0b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:22.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 shutdown_connections 2023-11-15T04:17:22.525 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:22.522+0000 7fc0bab91700 1 -- 172.21.15.188:0/128633257 wait complete. 2023-11-15T04:17:22.526 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:22.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:22 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:22.129393+0000 mgr.smithi038.bnbogz (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:23.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:23 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:22.521559+0000 mon.smithi038 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.188:0/128633257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:23.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:23.916 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:24.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:24 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:24.129771+0000 mgr.smithi038.bnbogz (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:25.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 -- 172.21.15.188:0/1147918667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 msgr2=0x7fdd040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:25.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 --2- 172.21.15.188:0/1147918667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd040f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fdcfc009fb0 tx=0x7fdcfc009b30).stop 2023-11-15T04:17:25.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 -- 172.21.15.188:0/1147918667 shutdown_connections 2023-11-15T04:17:25.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 --2- 172.21.15.188:0/1147918667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd040f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:25.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 -- 172.21.15.188:0/1147918667 >> 172.21.15.188:0/1147918667 conn(0x7fdd040f1c70 msgr2=0x7fdd040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:25.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 -- 172.21.15.188:0/1147918667 shutdown_connections 2023-11-15T04:17:25.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.177+0000 7fdd0c177700 1 -- 172.21.15.188:0/1147918667 wait complete. 2023-11-15T04:17:25.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd0c177700 1 Processor -- start 2023-11-15T04:17:25.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd0c177700 1 -- start start 2023-11-15T04:17:25.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd0c177700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:25.181 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd0c177700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04106c20 con 0x7fdd040f6510 2023-11-15T04:17:25.182 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd09f13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:25.182 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd09f13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56354/0 (socket says 172.21.15.188:56354) 2023-11-15T04:17:25.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.178+0000 7fdd09f13700 1 -- 172.21.15.188:0/2025057330 learned_addr learned my addr 172.21.15.188:0/2025057330 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:25.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdd09f13700 1 -- 172.21.15.188:0/2025057330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc005040 con 0x7fdd040f6510 2023-11-15T04:17:25.183 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdd09f13700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdcfc013510 tx=0x7fdcfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:25.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcfc009320 con 0x7fdd040f6510 2023-11-15T04:17:25.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04106e20 con 0x7fdd040f6510 2023-11-15T04:17:25.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcfc013f00 con 0x7fdd040f6510 2023-11-15T04:17:25.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcfc014d90 con 0x7fdd040f6510 2023-11-15T04:17:25.184 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.179+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd041072b0 con 0x7fdd040f6510 2023-11-15T04:17:25.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd04045ad0 con 0x7fdd040f6510 2023-11-15T04:17:25.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcfc0095f0 con 0x7fdd040f6510 2023-11-15T04:17:25.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdcfa7fc700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 0x7fdcf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:25.185 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcfc041000 con 0x7fdd040f6510 2023-11-15T04:17:25.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdd09712700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 0x7fdcf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:25.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.180+0000 7fdd09712700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 0x7fdcf0032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdcf40096a0 tx=0x7fdcf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:25.186 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.185+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcfc014420 con 0x7fdd040f6510 2023-11-15T04:17:25.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.388+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd040fbf60 con 0x7fdd040f6510 2023-11-15T04:17:25.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.389+0000 7fdcfa7fc700 1 -- 172.21.15.188:0/2025057330 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcfc01d7f0 con 0x7fdd040f6510 2023-11-15T04:17:25.389 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:25.390 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:25.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 msgr2=0x7fdcf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:25.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 0x7fdcf0032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdcf40096a0 tx=0x7fdcf4006b40).stop 2023-11-15T04:17:25.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 msgr2=0x7fdd04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:25.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdcfc013510 tx=0x7fdcfc00b040).stop 2023-11-15T04:17:25.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 shutdown_connections 2023-11-15T04:17:25.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fdcf002ff60 0x7fdcf0032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:25.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 --2- 172.21.15.188:0/2025057330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd040f6510 0x7fdd04106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:25.392 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 >> 172.21.15.188:0/2025057330 conn(0x7fdd040f1c70 msgr2=0x7fdd040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:25.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 shutdown_connections 2023-11-15T04:17:25.393 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:25.390+0000 7fdd0c177700 1 -- 172.21.15.188:0/2025057330 wait complete. 2023-11-15T04:17:25.393 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:25.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:25 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:25.389965+0000 mon.smithi038 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.188:0/2025057330' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:26.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:26.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:26.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:26 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:26.130191+0000 mgr.smithi038.bnbogz (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:28.058 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.057+0000 7f389b439700 1 -- 172.21.15.188:0/3798279074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 msgr2=0x7f38940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:28.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.057+0000 7f389b439700 1 --2- 172.21.15.188:0/3798279074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3884009fb0 tx=0x7f3884009b30).stop 2023-11-15T04:17:28.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.057+0000 7f389b439700 1 -- 172.21.15.188:0/3798279074 shutdown_connections 2023-11-15T04:17:28.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.057+0000 7f389b439700 1 --2- 172.21.15.188:0/3798279074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:28.059 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.057+0000 7f389b439700 1 -- 172.21.15.188:0/3798279074 >> 172.21.15.188:0/3798279074 conn(0x7f38940f0b70 msgr2=0x7f38940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:28.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 -- 172.21.15.188:0/3798279074 shutdown_connections 2023-11-15T04:17:28.060 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 -- 172.21.15.188:0/3798279074 wait complete. 2023-11-15T04:17:28.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 Processor -- start 2023-11-15T04:17:28.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 -- start start 2023-11-15T04:17:28.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:28.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.058+0000 7f389b439700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940fb820 con 0x7f38940f5410 2023-11-15T04:17:28.062 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f38991d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:28.063 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f38991d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:56372/0 (socket says 172.21.15.188:56372) 2023-11-15T04:17:28.063 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f38991d5700 1 -- 172.21.15.188:0/873461699 learned_addr learned my addr 172.21.15.188:0/873461699 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:28.063 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f38991d5700 1 -- 172.21.15.188:0/873461699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3884005040 con 0x7f38940f5410 2023-11-15T04:17:28.063 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f38991d5700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3884013510 tx=0x7f388400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:28.063 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3884009320 con 0x7f38940f5410 2023-11-15T04:17:28.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38940fb9c0 con 0x7f38940f5410 2023-11-15T04:17:28.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3884013f00 con 0x7f38940f5410 2023-11-15T04:17:28.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3884014d90 con 0x7f38940f5410 2023-11-15T04:17:28.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.059+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38940fbe50 con 0x7f38940f5410 2023-11-15T04:17:28.064 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.060+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3894059100 con 0x7f38940f5410 2023-11-15T04:17:28.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.060+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38840095f0 con 0x7f38940f5410 2023-11-15T04:17:28.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.060+0000 7f3889ffb700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 0x7f3880032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:28.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.060+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3884040fa0 con 0x7f38940f5410 2023-11-15T04:17:28.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.061+0000 7f38989d4700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 0x7f3880032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:28.065 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.061+0000 7f38989d4700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 0x7f3880032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f38900096a0 tx=0x7f3890006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:28.066 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.065+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f388406d2b0 con 0x7f38940f5410 2023-11-15T04:17:28.274 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.271+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38940fae60 con 0x7f38940f5410 2023-11-15T04:17:28.274 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.272+0000 7f3889ffb700 1 -- 172.21.15.188:0/873461699 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3884014420 con 0x7f38940f5410 2023-11-15T04:17:28.274 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:28.275 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:28.275 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 msgr2=0x7f3880032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:28.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 0x7f3880032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f38900096a0 tx=0x7f3890006b40).stop 2023-11-15T04:17:28.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 msgr2=0x7f38940fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:28.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3884013510 tx=0x7f388400b040).stop 2023-11-15T04:17:28.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 shutdown_connections 2023-11-15T04:17:28.276 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f388002ffb0 0x7f3880032470 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:28.277 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 --2- 172.21.15.188:0/873461699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38940f5410 0x7f38940fd3a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:28.277 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 >> 172.21.15.188:0/873461699 conn(0x7f38940f0b70 msgr2=0x7f389405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:28.277 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 shutdown_connections 2023-11-15T04:17:28.277 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:28.274+0000 7f389b439700 1 -- 172.21.15.188:0/873461699 wait complete. 2023-11-15T04:17:28.277 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:28.130595+0000 mgr.smithi038.bnbogz (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:29.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:29 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:28.273651+0000 mon.smithi038 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.188:0/873461699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:29.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:29.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:30.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/1419421799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 msgr2=0x7fd8a00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:30.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/1419421799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f7d00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fd890009fb0 tx=0x7fd890009b30).stop 2023-11-15T04:17:30.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/1419421799 shutdown_connections 2023-11-15T04:17:30.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/1419421799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f7d00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:30.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/1419421799 >> 172.21.15.188:0/1419421799 conn(0x7fd8a00f0120 msgr2=0x7fd8a00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:30.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/1419421799 shutdown_connections 2023-11-15T04:17:30.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/1419421799 wait complete. 2023-11-15T04:17:30.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 Processor -- start 2023-11-15T04:17:30.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.978+0000 7fd8a4b1f700 1 -- start start 2023-11-15T04:17:30.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd8a4b1f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:30.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd8a4b1f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00f5630 con 0x7fd8a00f78e0 2023-11-15T04:17:30.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd89f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:30.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd89f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:40978/0 (socket says 172.21.15.188:40978) 2023-11-15T04:17:30.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd89f7fe700 1 -- 172.21.15.188:0/2546001174 learned_addr learned my addr 172.21.15.188:0/2546001174 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd89f7fe700 1 -- 172.21.15.188:0/2546001174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890005040 con 0x7fd8a00f78e0 2023-11-15T04:17:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.979+0000 7fd89f7fe700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fd890013510 tx=0x7fd89000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd890009320 con 0x7fd8a00f78e0 2023-11-15T04:17:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a00f57d0 con 0x7fd8a00f78e0 2023-11-15T04:17:30.984 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd890013f00 con 0x7fd8a00f78e0 2023-11-15T04:17:30.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd890014dc0 con 0x7fd8a00f78e0 2023-11-15T04:17:30.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a00f5c60 con 0x7fd8a00f78e0 2023-11-15T04:17:30.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.980+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a0059100 con 0x7fd8a00f78e0 2023-11-15T04:17:30.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.981+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8900095f0 con 0x7fd8a00f78e0 2023-11-15T04:17:30.985 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.981+0000 7fd887fff700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 0x7fd888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:30.986 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.981+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd890041000 con 0x7fd8a00f78e0 2023-11-15T04:17:30.986 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.981+0000 7fd89effd700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 0x7fd888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:30.986 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.981+0000 7fd89effd700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 0x7fd888032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd8940096a0 tx=0x7fd894006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:30.986 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:30.986+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd890014420 con 0x7fd8a00f78e0 2023-11-15T04:17:31.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.190+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8a00fbce0 con 0x7fd8a00f78e0 2023-11-15T04:17:31.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.191+0000 7fd887fff700 1 -- 172.21.15.188:0/2546001174 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd890010230 con 0x7fd8a00f78e0 2023-11-15T04:17:31.192 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:31.192 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:31.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 msgr2=0x7fd888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:31.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 0x7fd888032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd8940096a0 tx=0x7fd894006b40).stop 2023-11-15T04:17:31.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 msgr2=0x7fd8a00f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:31.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fd890013510 tx=0x7fd89000b040).stop 2023-11-15T04:17:31.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 shutdown_connections 2023-11-15T04:17:31.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.192+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fd88802ff60 0x7fd888032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:31.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.193+0000 7fd8a4b1f700 1 --2- 172.21.15.188:0/2546001174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8a00f78e0 0x7fd8a00f71b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:31.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.193+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 >> 172.21.15.188:0/2546001174 conn(0x7fd8a00f0120 msgr2=0x7fd8a00f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:31.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.193+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 shutdown_connections 2023-11-15T04:17:31.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:31.193+0000 7fd8a4b1f700 1 -- 172.21.15.188:0/2546001174 wait complete. 2023-11-15T04:17:31.196 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:31.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:31 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:30.130948+0000 mgr.smithi038.bnbogz (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:32.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:32.563 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:32.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:32 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:31.192177+0000 mon.smithi038 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.188:0/2546001174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:33.594 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:33 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:32.131310+0000 mgr.smithi038.bnbogz (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:33.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 -- 172.21.15.188:0/2093793189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 msgr2=0x7f45600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:33.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 --2- 172.21.15.188:0/2093793189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f45600f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4548009fb0 tx=0x7f4548009b30).stop 2023-11-15T04:17:33.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 -- 172.21.15.188:0/2093793189 shutdown_connections 2023-11-15T04:17:33.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 --2- 172.21.15.188:0/2093793189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f45600f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:33.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 -- 172.21.15.188:0/2093793189 >> 172.21.15.188:0/2093793189 conn(0x7f45600f1c70 msgr2=0x7f45600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:33.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 -- 172.21.15.188:0/2093793189 shutdown_connections 2023-11-15T04:17:33.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.778+0000 7f4565839700 1 -- 172.21.15.188:0/2093793189 wait complete. 2023-11-15T04:17:33.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.779+0000 7f4565839700 1 Processor -- start 2023-11-15T04:17:33.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.779+0000 7f4565839700 1 -- start start 2023-11-15T04:17:33.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.779+0000 7f4565839700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:33.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.779+0000 7f4565839700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560106c20 con 0x7f45600f6510 2023-11-15T04:17:33.783 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f455effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:33.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f455effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:41004/0 (socket says 172.21.15.188:41004) 2023-11-15T04:17:33.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f455effd700 1 -- 172.21.15.188:0/3972911898 learned_addr learned my addr 172.21.15.188:0/3972911898 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:33.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f455effd700 1 -- 172.21.15.188:0/3972911898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548005040 con 0x7f45600f6510 2023-11-15T04:17:33.784 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f455effd700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f4548013510 tx=0x7f454800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:33.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4548009320 con 0x7f45600f6510 2023-11-15T04:17:33.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560106e20 con 0x7f45600f6510 2023-11-15T04:17:33.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4548013f00 con 0x7f45600f6510 2023-11-15T04:17:33.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4548014d90 con 0x7f45600f6510 2023-11-15T04:17:33.785 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.780+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45601072b0 con 0x7f45600f6510 2023-11-15T04:17:33.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.781+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4548014420 con 0x7f45600f6510 2023-11-15T04:17:33.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.781+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560045ad0 con 0x7f45600f6510 2023-11-15T04:17:33.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.781+0000 7f45577fe700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 0x7f454c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:33.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.781+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4548040fb0 con 0x7f45600f6510 2023-11-15T04:17:33.786 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.782+0000 7f455e7fc700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 0x7f454c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:33.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.782+0000 7f455e7fc700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 0x7f454c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f45500096a0 tx=0x7f4550006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:33.787 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.786+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45480095f0 con 0x7f45600f6510 2023-11-15T04:17:33.992 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.991+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45600fbf60 con 0x7f45600f6510 2023-11-15T04:17:33.992 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.991+0000 7f45577fe700 1 -- 172.21.15.188:0/3972911898 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f454801db70 con 0x7f45600f6510 2023-11-15T04:17:33.993 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:33.993 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:33.994 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 msgr2=0x7f454c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:33.995 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 0x7f454c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f45500096a0 tx=0x7f4550006b40).stop 2023-11-15T04:17:33.995 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 msgr2=0x7f4560106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:33.995 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f4548013510 tx=0x7f454800b040).stop 2023-11-15T04:17:33.995 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 shutdown_connections 2023-11-15T04:17:33.996 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f454c02ff60 0x7f454c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:33.996 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 --2- 172.21.15.188:0/3972911898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45600f6510 0x7f4560106730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:33.996 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 >> 172.21.15.188:0/3972911898 conn(0x7f45600f1c70 msgr2=0x7f45600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:33.996 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 shutdown_connections 2023-11-15T04:17:33.996 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:33.993+0000 7f4565839700 1 -- 172.21.15.188:0/3972911898 wait complete. 2023-11-15T04:17:33.997 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:34.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:34 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:33.992974+0000 mon.smithi038 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.188:0/3972911898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:35.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:35.459 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:35.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:35 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:34.131664+0000 mgr.smithi038.bnbogz (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:36.770 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.766+0000 7f308e545700 1 -- 172.21.15.188:0/861946782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 msgr2=0x7f30880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:36.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.766+0000 7f308e545700 1 --2- 172.21.15.188:0/861946782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f30880f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f3078009fb0 tx=0x7f3078009b30).stop 2023-11-15T04:17:36.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- 172.21.15.188:0/861946782 shutdown_connections 2023-11-15T04:17:36.771 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 --2- 172.21.15.188:0/861946782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f30880f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:36.772 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- 172.21.15.188:0/861946782 >> 172.21.15.188:0/861946782 conn(0x7f30880f0b70 msgr2=0x7f30880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:36.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- 172.21.15.188:0/861946782 shutdown_connections 2023-11-15T04:17:36.773 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- 172.21.15.188:0/861946782 wait complete. 2023-11-15T04:17:36.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 Processor -- start 2023-11-15T04:17:36.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- start start 2023-11-15T04:17:36.774 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:36.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.767+0000 7f308e545700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088106c00 con 0x7f30880f5410 2023-11-15T04:17:36.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3087fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:36.775 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3087fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:41018/0 (socket says 172.21.15.188:41018) 2023-11-15T04:17:36.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3087fff700 1 -- 172.21.15.188:0/2812520756 learned_addr learned my addr 172.21.15.188:0/2812520756 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:36.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3087fff700 1 -- 172.21.15.188:0/2812520756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078005040 con 0x7f30880f5410 2023-11-15T04:17:36.776 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3087fff700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3078014000 tx=0x7f307800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:36.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30780095f0 con 0x7f30880f5410 2023-11-15T04:17:36.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3088106e00 con 0x7f30880f5410 2023-11-15T04:17:36.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3078007e70 con 0x7f30880f5410 2023-11-15T04:17:36.777 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3078014ea0 con 0x7f30880f5410 2023-11-15T04:17:36.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.768+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3088107290 con 0x7f30880f5410 2023-11-15T04:17:36.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.769+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3078025070 con 0x7f30880f5410 2023-11-15T04:17:36.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.769+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3088045ad0 con 0x7f30880f5410 2023-11-15T04:17:36.778 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.769+0000 7f3084ff9700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 0x7f3070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:36.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.769+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30780409b0 con 0x7f30880f5410 2023-11-15T04:17:36.779 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.770+0000 7f30877fe700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 0x7f3070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:36.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.770+0000 7f30877fe700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 0x7f3070032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f307c0096a0 tx=0x7f307c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:36.780 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.774+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30780079c0 con 0x7f30880f5410 2023-11-15T04:17:36.979 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.978+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30880fae60 con 0x7f30880f5410 2023-11-15T04:17:36.980 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.979+0000 7f3084ff9700 1 -- 172.21.15.188:0/2812520756 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f307800f300 con 0x7f30880f5410 2023-11-15T04:17:36.980 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:36.980 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:36.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 msgr2=0x7f3070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:36.981 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 0x7f3070032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f307c0096a0 tx=0x7f307c006b40).stop 2023-11-15T04:17:36.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 msgr2=0x7f3088106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:36.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3078014000 tx=0x7f307800b040).stop 2023-11-15T04:17:36.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 shutdown_connections 2023-11-15T04:17:36.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f307002ff60 0x7f3070032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:36.982 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 --2- 172.21.15.188:0/2812520756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30880f5410 0x7f3088106710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:36.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 >> 172.21.15.188:0/2812520756 conn(0x7f30880f0b70 msgr2=0x7f308805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:36.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.980+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 shutdown_connections 2023-11-15T04:17:36.983 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:36.981+0000 7f308e545700 1 -- 172.21.15.188:0/2812520756 wait complete. 2023-11-15T04:17:36.983 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:36.132121+0000 mgr.smithi038.bnbogz (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:37.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:37 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:36.980254+0000 mon.smithi038 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.188:0/2812520756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:38.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:38.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:39.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:39 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:38.132595+0000 mgr.smithi038.bnbogz (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:39.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 -- 172.21.15.188:0/2600532446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 msgr2=0x7f14f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:39.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 --2- 172.21.15.188:0/2600532446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f40f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f14e8009fb0 tx=0x7f14e8009b30).stop 2023-11-15T04:17:39.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 -- 172.21.15.188:0/2600532446 shutdown_connections 2023-11-15T04:17:39.618 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 --2- 172.21.15.188:0/2600532446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f40f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:39.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 -- 172.21.15.188:0/2600532446 >> 172.21.15.188:0/2600532446 conn(0x7f14f40f0b70 msgr2=0x7f14f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:39.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 -- 172.21.15.188:0/2600532446 shutdown_connections 2023-11-15T04:17:39.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.616+0000 7f14f8c73700 1 -- 172.21.15.188:0/2600532446 wait complete. 2023-11-15T04:17:39.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f8c73700 1 Processor -- start 2023-11-15T04:17:39.619 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f8c73700 1 -- start start 2023-11-15T04:17:39.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f8c73700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:39.620 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f8c73700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4108e00 con 0x7f14f40f5410 2023-11-15T04:17:39.621 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:39.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:35620/0 (socket says 172.21.15.188:35620) 2023-11-15T04:17:39.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.617+0000 7f14f259c700 1 -- 172.21.15.188:0/343583083 learned_addr learned my addr 172.21.15.188:0/343583083 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:39.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14f259c700 1 -- 172.21.15.188:0/343583083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8005040 con 0x7f14f40f5410 2023-11-15T04:17:39.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14f259c700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f14e8013510 tx=0x7f14e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:39.622 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e8009320 con 0x7f14f40f5410 2023-11-15T04:17:39.623 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f4109000 con 0x7f14f40f5410 2023-11-15T04:17:39.623 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14e8013f00 con 0x7f14f40f5410 2023-11-15T04:17:39.623 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14e8014d90 con 0x7f14f40f5410 2023-11-15T04:17:39.623 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.618+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f4109490 con 0x7f14f40f5410 2023-11-15T04:17:39.623 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.619+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f4045ad0 con 0x7f14f40f5410 2023-11-15T04:17:39.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.619+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14e8014420 con 0x7f14f40f5410 2023-11-15T04:17:39.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.619+0000 7f14e2ffd700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:39.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.619+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14e80410c0 con 0x7f14f40f5410 2023-11-15T04:17:39.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.619+0000 7f14f1d9b700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:39.624 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.620+0000 7f14f1d9b700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 0x7f14dc032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:39.625 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.624+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14e80095f0 con 0x7f14f40f5410 2023-11-15T04:17:39.831 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.831+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14f40fae60 con 0x7f14f40f5410 2023-11-15T04:17:39.832 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.831+0000 7f14e2ffd700 1 -- 172.21.15.188:0/343583083 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14e801db70 con 0x7f14f40f5410 2023-11-15T04:17:39.833 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:39.833 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:39.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 msgr2=0x7f14dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:39.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 0x7f14dc032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).stop 2023-11-15T04:17:39.834 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 msgr2=0x7f14f4108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:39.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f14e8013510 tx=0x7f14e800b040).stop 2023-11-15T04:17:39.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 shutdown_connections 2023-11-15T04:17:39.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f14dc02ff60 0x7f14dc032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:39.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 --2- 172.21.15.188:0/343583083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14f40f5410 0x7f14f4108910 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:39.835 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 >> 172.21.15.188:0/343583083 conn(0x7f14f40f0b70 msgr2=0x7f14f405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:39.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 shutdown_connections 2023-11-15T04:17:39.836 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:39.833+0000 7f14f8c73700 1 -- 172.21.15.188:0/343583083 wait complete. 2023-11-15T04:17:39.836 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:40.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:40 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:39.832871+0000 mon.smithi038 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.188:0/343583083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:41.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:41.228 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:41.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:41 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:40.132996+0000 mgr.smithi038.bnbogz (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:42.451 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 -- 172.21.15.188:0/1121796492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 msgr2=0x7fadb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:42.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/1121796492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb40f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fada4009fb0 tx=0x7fada4009b30).stop 2023-11-15T04:17:42.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 -- 172.21.15.188:0/1121796492 shutdown_connections 2023-11-15T04:17:42.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/1121796492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb40f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:42.452 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 -- 172.21.15.188:0/1121796492 >> 172.21.15.188:0/1121796492 conn(0x7fadb40f1c70 msgr2=0x7fadb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:42.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 -- 172.21.15.188:0/1121796492 shutdown_connections 2023-11-15T04:17:42.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.449+0000 7fadb8b9a700 1 -- 172.21.15.188:0/1121796492 wait complete. 2023-11-15T04:17:42.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb8b9a700 1 Processor -- start 2023-11-15T04:17:42.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb8b9a700 1 -- start start 2023-11-15T04:17:42.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb8b9a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:42.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb8b9a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4106c20 con 0x7fadb40f6510 2023-11-15T04:17:42.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:42.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:35638/0 (socket says 172.21.15.188:35638) 2023-11-15T04:17:42.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb259c700 1 -- 172.21.15.188:0/2665641564 learned_addr learned my addr 172.21.15.188:0/2665641564 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:42.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb259c700 1 -- 172.21.15.188:0/2665641564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4005040 con 0x7fadb40f6510 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.450+0000 7fadb259c700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fada4013510 tx=0x7fada400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada4009320 con 0x7fadb40f6510 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4106e20 con 0x7fadb40f6510 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fada4013f00 con 0x7fadb40f6510 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada4014d90 con 0x7fadb40f6510 2023-11-15T04:17:42.455 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb41072b0 con 0x7fadb40f6510 2023-11-15T04:17:42.456 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.451+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fada40263a0 con 0x7fadb40f6510 2023-11-15T04:17:42.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.452+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb4045ad0 con 0x7fadb40f6510 2023-11-15T04:17:42.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.452+0000 7fada2ffd700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 0x7fad9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:42.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.452+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fada40413c0 con 0x7fadb40f6510 2023-11-15T04:17:42.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.452+0000 7fadb1d9b700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 0x7fad9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:42.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.452+0000 7fadb1d9b700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 0x7fad9c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:42.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.457+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fada4012920 con 0x7fadb40f6510 2023-11-15T04:17:42.662 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.662+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadb40fbf60 con 0x7fadb40f6510 2023-11-15T04:17:42.663 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.662+0000 7fada2ffd700 1 -- 172.21.15.188:0/2665641564 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fada4012330 con 0x7fadb40f6510 2023-11-15T04:17:42.663 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:42.664 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:42.666 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 msgr2=0x7fad9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:42.666 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 0x7fad9c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).stop 2023-11-15T04:17:42.666 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 msgr2=0x7fadb4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:42.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fada4013510 tx=0x7fada400b040).stop 2023-11-15T04:17:42.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 shutdown_connections 2023-11-15T04:17:42.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fad9c02ff60 0x7fad9c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:42.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 --2- 172.21.15.188:0/2665641564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:42.667 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 >> 172.21.15.188:0/2665641564 conn(0x7fadb40f1c70 msgr2=0x7fadb40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:42.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 shutdown_connections 2023-11-15T04:17:42.668 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:42.664+0000 7fadb8b9a700 1 -- 172.21.15.188:0/2665641564 wait complete. 2023-11-15T04:17:42.668 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:42.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:42 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:42.133398+0000 mgr.smithi038.bnbogz (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:43.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:43 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:42.663934+0000 mon.smithi038 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.188:0/2665641564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:44.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:44.095 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:44.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:44 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:44.133794+0000 mgr.smithi038.bnbogz (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 -- 172.21.15.188:0/4134295024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 msgr2=0x7f23c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 --2- 172.21.15.188:0/4134295024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c40f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f23b4009fb0 tx=0x7f23b4009b30).stop 2023-11-15T04:17:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 -- 172.21.15.188:0/4134295024 shutdown_connections 2023-11-15T04:17:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 --2- 172.21.15.188:0/4134295024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c40f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:45.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 -- 172.21.15.188:0/4134295024 >> 172.21.15.188:0/4134295024 conn(0x7f23c40f0b70 msgr2=0x7f23c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:45.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.382+0000 7f23cb564700 1 -- 172.21.15.188:0/4134295024 shutdown_connections 2023-11-15T04:17:45.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23cb564700 1 -- 172.21.15.188:0/4134295024 wait complete. 2023-11-15T04:17:45.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23cb564700 1 Processor -- start 2023-11-15T04:17:45.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23cb564700 1 -- start start 2023-11-15T04:17:45.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23cb564700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:45.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23cb564700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4107240 con 0x7f23c40f5410 2023-11-15T04:17:45.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23c9300700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:45.387 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23c9300700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:35646/0 (socket says 172.21.15.188:35646) 2023-11-15T04:17:45.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.383+0000 7f23c9300700 1 -- 172.21.15.188:0/2989354499 learned_addr learned my addr 172.21.15.188:0/2989354499 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:45.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23c9300700 1 -- 172.21.15.188:0/2989354499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b4005040 con 0x7f23c40f5410 2023-11-15T04:17:45.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23c9300700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f23b4013510 tx=0x7f23b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:45.388 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b4009320 con 0x7f23c40f5410 2023-11-15T04:17:45.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c4060220 con 0x7f23c40f5410 2023-11-15T04:17:45.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23b4013f00 con 0x7f23c40f5410 2023-11-15T04:17:45.389 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b4014d90 con 0x7f23c40f5410 2023-11-15T04:17:45.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.384+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c410bd40 con 0x7f23c40f5410 2023-11-15T04:17:45.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.385+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c4045ad0 con 0x7f23c40f5410 2023-11-15T04:17:45.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.385+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23b40095f0 con 0x7f23c40f5410 2023-11-15T04:17:45.390 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.385+0000 7f23b9ffb700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 0x7f23b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:45.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.385+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23b4041000 con 0x7f23c40f5410 2023-11-15T04:17:45.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.386+0000 7f23c8aff700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 0x7f23b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:45.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.386+0000 7f23c8aff700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 0x7f23b0032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f23c00096a0 tx=0x7f23c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:45.391 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.390+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23b401d7f0 con 0x7f23c40f5410 2023-11-15T04:17:45.595 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.595+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23c40fae60 con 0x7f23c40f5410 2023-11-15T04:17:45.596 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.596+0000 7f23b9ffb700 1 -- 172.21.15.188:0/2989354499 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23b4010230 con 0x7f23c40f5410 2023-11-15T04:17:45.597 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:45.597 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:45.598 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 msgr2=0x7f23b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:45.598 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 0x7f23b0032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f23c00096a0 tx=0x7f23c0006b40).stop 2023-11-15T04:17:45.599 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 msgr2=0x7f23c410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:45.599 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f23b4013510 tx=0x7f23b400b040).stop 2023-11-15T04:17:45.599 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 shutdown_connections 2023-11-15T04:17:45.599 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f23b002ff60 0x7f23b0032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:45.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 --2- 172.21.15.188:0/2989354499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23c40f5410 0x7f23c410b850 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:45.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 >> 172.21.15.188:0/2989354499 conn(0x7f23c40f0b70 msgr2=0x7f23c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:45.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 shutdown_connections 2023-11-15T04:17:45.600 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:45.597+0000 7f23cb564700 1 -- 172.21.15.188:0/2989354499 wait complete. 2023-11-15T04:17:45.601 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:46.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:45 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:45.597063+0000 mon.smithi038 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.188:0/2989354499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:46.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:46.996 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:47.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:46 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:46.134172+0000 mgr.smithi038.bnbogz (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:48.282 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/850202285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 msgr2=0x7f5fb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:48.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/850202285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb40f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5fa4009fb0 tx=0x7f5fa4009b30).stop 2023-11-15T04:17:48.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/850202285 shutdown_connections 2023-11-15T04:17:48.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/850202285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb40f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:48.283 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/850202285 >> 172.21.15.188:0/850202285 conn(0x7f5fb40f0b70 msgr2=0x7f5fb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:48.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/850202285 shutdown_connections 2023-11-15T04:17:48.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.280+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/850202285 wait complete. 2023-11-15T04:17:48.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.281+0000 7f5fb9e8b700 1 Processor -- start 2023-11-15T04:17:48.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.281+0000 7f5fb9e8b700 1 -- start start 2023-11-15T04:17:48.284 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.281+0000 7f5fb9e8b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.281+0000 7f5fb9e8b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb4105b20 con 0x7f5fb40f5410 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:35660/0 (socket says 172.21.15.188:35660) 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb37fe700 1 -- 172.21.15.188:0/2336884235 learned_addr learned my addr 172.21.15.188:0/2336884235 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb37fe700 1 -- 172.21.15.188:0/2336884235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa4005040 con 0x7f5fb40f5410 2023-11-15T04:17:48.285 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb37fe700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5fa4014000 tx=0x7f5fa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:48.286 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa40095f0 con 0x7f5fb40f5410 2023-11-15T04:17:48.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fa4007e70 con 0x7f5fb40f5410 2023-11-15T04:17:48.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb4105d20 con 0x7f5fb40f5410 2023-11-15T04:17:48.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa4014ea0 con 0x7f5fb40f5410 2023-11-15T04:17:48.287 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.282+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb41061b0 con 0x7f5fb40f5410 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.283+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fa4025070 con 0x7f5fb40f5410 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.283+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb4045ad0 con 0x7f5fb40f5410 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.283+0000 7f5fabfff700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 0x7f5fa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.283+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fa4040970 con 0x7f5fb40f5410 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.283+0000 7f5fb2ffd700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 0x7f5fa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:48.288 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.284+0000 7f5fb2ffd700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 0x7f5fa0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5f9c0096a0 tx=0x7f5f9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:48.289 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.289+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fa40079c0 con 0x7f5fb40f5410 2023-11-15T04:17:48.494 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.493+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fb40fae60 con 0x7f5fb40f5410 2023-11-15T04:17:48.495 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.494+0000 7f5fabfff700 1 -- 172.21.15.188:0/2336884235 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fa400f300 con 0x7f5fb40f5410 2023-11-15T04:17:48.495 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:48.495 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:48.496 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 msgr2=0x7f5fa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:48.497 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 0x7f5fa0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5f9c0096a0 tx=0x7f5f9c006b40).stop 2023-11-15T04:17:48.497 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 msgr2=0x7f5fb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:48.497 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5fa4014000 tx=0x7f5fa400b040).stop 2023-11-15T04:17:48.497 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 shutdown_connections 2023-11-15T04:17:48.497 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f5fa002ff60 0x7f5fa0032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:48.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 --2- 172.21.15.188:0/2336884235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fb40f5410 0x7f5fb4105630 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:48.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 >> 172.21.15.188:0/2336884235 conn(0x7f5fb40f0b70 msgr2=0x7f5fb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:48.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 shutdown_connections 2023-11-15T04:17:48.498 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:48.496+0000 7f5fb9e8b700 1 -- 172.21.15.188:0/2336884235 wait complete. 2023-11-15T04:17:48.499 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:48.134476+0000 mgr.smithi038.bnbogz (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:49.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:49 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:48.495455+0000 mon.smithi038 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.188:0/2336884235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:49.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:49.891 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:51.172 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- 172.21.15.188:0/1387437813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 msgr2=0x7f70a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:51.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/1387437813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a40f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f7094009fb0 tx=0x7f7094009b30).stop 2023-11-15T04:17:51.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- 172.21.15.188:0/1387437813 shutdown_connections 2023-11-15T04:17:51.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/1387437813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a40f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:51.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- 172.21.15.188:0/1387437813 >> 172.21.15.188:0/1387437813 conn(0x7f70a40f0b70 msgr2=0x7f70a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:51.173 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- 172.21.15.188:0/1387437813 shutdown_connections 2023-11-15T04:17:51.174 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- 172.21.15.188:0/1387437813 wait complete. 2023-11-15T04:17:51.174 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 Processor -- start 2023-11-15T04:17:51.174 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.172+0000 7f70ac3fe700 1 -- start start 2023-11-15T04:17:51.174 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70ac3fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:51.174 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70ac3fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a4060220 con 0x7f70a40f5410 2023-11-15T04:17:51.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70aa19a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:51.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70aa19a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46890/0 (socket says 172.21.15.188:46890) 2023-11-15T04:17:51.176 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70aa19a700 1 -- 172.21.15.188:0/246724686 learned_addr learned my addr 172.21.15.188:0/246724686 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:51.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70aa19a700 1 -- 172.21.15.188:0/246724686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094005040 con 0x7f70a40f5410 2023-11-15T04:17:51.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.173+0000 7f70aa19a700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7094013510 tx=0x7f709400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:51.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.174+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7094009320 con 0x7f70a40f5410 2023-11-15T04:17:51.177 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.174+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a4060420 con 0x7f70a40f5410 2023-11-15T04:17:51.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.174+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7094013f00 con 0x7f70a40f5410 2023-11-15T04:17:51.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.174+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7094014d90 con 0x7f70a40f5410 2023-11-15T04:17:51.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.174+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a4109b00 con 0x7f70a40f5410 2023-11-15T04:17:51.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a4059100 con 0x7f70a40f5410 2023-11-15T04:17:51.178 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70940095f0 con 0x7f70a40f5410 2023-11-15T04:17:51.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f709affd700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 0x7f7090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:51.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7094040fe0 con 0x7f70a40f5410 2023-11-15T04:17:51.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f70a9999700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 0x7f7090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:51.179 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.175+0000 7f70a9999700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 0x7f7090032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f70a00096a0 tx=0x7f70a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:51.180 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.180+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7094014420 con 0x7f70a40f5410 2023-11-15T04:17:51.382 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.382+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70a4045ad0 con 0x7f70a40f5410 2023-11-15T04:17:51.383 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.383+0000 7f709affd700 1 -- 172.21.15.188:0/246724686 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7094010230 con 0x7f70a40f5410 2023-11-15T04:17:51.383 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:51.383 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:51.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 msgr2=0x7f7090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:51.384 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 0x7f7090032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f70a00096a0 tx=0x7f70a0006b40).stop 2023-11-15T04:17:51.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 msgr2=0x7f70a4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:51.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7094013510 tx=0x7f709400b040).stop 2023-11-15T04:17:51.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 shutdown_connections 2023-11-15T04:17:51.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f709002ff60 0x7f7090032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:51.385 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 --2- 172.21.15.188:0/246724686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a40f5410 0x7f70a4109450 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:51.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.384+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 >> 172.21.15.188:0/246724686 conn(0x7f70a40f0b70 msgr2=0x7f70a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:51.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.385+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 shutdown_connections 2023-11-15T04:17:51.386 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:51.385+0000 7f70ac3fe700 1 -- 172.21.15.188:0/246724686 wait complete. 2023-11-15T04:17:51.386 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:50.134905+0000 mgr.smithi038.bnbogz (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:50.216315+0000 mon.smithi038 (mon.0) 321 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:17:51.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:51 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:50.217457+0000 mon.smithi038 (mon.0) 322 : audit [INF] from='mgr.14164 172.21.15.38:0/2988798439' entity='mgr.smithi038.bnbogz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi038.bnbogz/trash_purge_schedule"}]: dispatch 2023-11-15T04:17:52.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:52 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:51.383399+0000 mon.smithi038 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.188:0/246724686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:52.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:52.884 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:53.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:53 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:52.135413+0000 mgr.smithi038.bnbogz (mgr.14164) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:54.187 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 -- 172.21.15.188:0/600281720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 msgr2=0x7f131c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:54.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 --2- 172.21.15.188:0/600281720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c0f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f130c009fb0 tx=0x7f130c009b30).stop 2023-11-15T04:17:54.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 -- 172.21.15.188:0/600281720 shutdown_connections 2023-11-15T04:17:54.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 --2- 172.21.15.188:0/600281720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c0f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:54.188 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 -- 172.21.15.188:0/600281720 >> 172.21.15.188:0/600281720 conn(0x7f131c0f0b70 msgr2=0x7f131c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:54.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 -- 172.21.15.188:0/600281720 shutdown_connections 2023-11-15T04:17:54.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.187+0000 7f1322286700 1 -- 172.21.15.188:0/600281720 wait complete. 2023-11-15T04:17:54.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f1322286700 1 Processor -- start 2023-11-15T04:17:54.189 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f1322286700 1 -- start start 2023-11-15T04:17:54.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f1322286700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:54.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f1322286700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c060240 con 0x7f131c0f5410 2023-11-15T04:17:54.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f131bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:54.191 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f131bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46898/0 (socket says 172.21.15.188:46898) 2023-11-15T04:17:54.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.188+0000 7f131bfff700 1 -- 172.21.15.188:0/1964425250 learned_addr learned my addr 172.21.15.188:0/1964425250 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:54.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f131bfff700 1 -- 172.21.15.188:0/1964425250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c005040 con 0x7f131c0f5410 2023-11-15T04:17:54.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f131bfff700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f130c013510 tx=0x7f130c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:54.192 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c009320 con 0x7f131c0f5410 2023-11-15T04:17:54.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c060440 con 0x7f131c0f5410 2023-11-15T04:17:54.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f130c013f00 con 0x7f131c0f5410 2023-11-15T04:17:54.193 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c014d90 con 0x7f131c0f5410 2023-11-15T04:17:54.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.189+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c109ab0 con 0x7f131c0f5410 2023-11-15T04:17:54.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.190+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f130c0095f0 con 0x7f131c0f5410 2023-11-15T04:17:54.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.190+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c045ad0 con 0x7f131c0f5410 2023-11-15T04:17:54.194 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.190+0000 7f1318ff9700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 0x7f1304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:54.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.190+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f130c041000 con 0x7f131c0f5410 2023-11-15T04:17:54.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.190+0000 7f131b7fe700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 0x7f1304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:54.195 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.191+0000 7f131b7fe700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 0x7f1304032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f13100096a0 tx=0x7f1310006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:54.196 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.195+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f130c01d7f0 con 0x7f131c0f5410 2023-11-15T04:17:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.402+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f131c0fae60 con 0x7f131c0f5410 2023-11-15T04:17:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.403+0000 7f1318ff9700 1 -- 172.21.15.188:0/1964425250 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f130c010230 con 0x7f131c0f5410 2023-11-15T04:17:54.403 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:54.403 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:54.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 msgr2=0x7f1304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:54.404 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 0x7f1304032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f13100096a0 tx=0x7f1310006b40).stop 2023-11-15T04:17:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 msgr2=0x7f131c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f130c013510 tx=0x7f130c00b040).stop 2023-11-15T04:17:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 shutdown_connections 2023-11-15T04:17:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f130402ff60 0x7f1304032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 --2- 172.21.15.188:0/1964425250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f131c0f5410 0x7f131c109400 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.404+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 >> 172.21.15.188:0/1964425250 conn(0x7f131c0f0b70 msgr2=0x7f131c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.405+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 shutdown_connections 2023-11-15T04:17:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:54.405+0000 7f1322286700 1 -- 172.21.15.188:0/1964425250 wait complete. 2023-11-15T04:17:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:54.135844+0000 mgr.smithi038.bnbogz (mgr.14164) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:55.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:55 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:54.403206+0000 mon.smithi038 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.188:0/1964425250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:55.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:55.750 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:57.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 -- 172.21.15.188:0/3845802237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 msgr2=0x7fb0800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:57.019 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 --2- 172.21.15.188:0/3845802237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb0800f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fb074009fb0 tx=0x7fb074009b30).stop 2023-11-15T04:17:57.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 -- 172.21.15.188:0/3845802237 shutdown_connections 2023-11-15T04:17:57.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 --2- 172.21.15.188:0/3845802237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb0800f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:57.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 -- 172.21.15.188:0/3845802237 >> 172.21.15.188:0/3845802237 conn(0x7fb0800f1c70 msgr2=0x7fb0800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:57.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 -- 172.21.15.188:0/3845802237 shutdown_connections 2023-11-15T04:17:57.020 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.018+0000 7fb086af2700 1 -- 172.21.15.188:0/3845802237 wait complete. 2023-11-15T04:17:57.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb086af2700 1 Processor -- start 2023-11-15T04:17:57.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb086af2700 1 -- start start 2023-11-15T04:17:57.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb086af2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:57.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb086af2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb080106c20 con 0x7fb0800f6510 2023-11-15T04:17:57.021 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb08488e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:57.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb08488e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:46908/0 (socket says 172.21.15.188:46908) 2023-11-15T04:17:57.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.019+0000 7fb08488e700 1 -- 172.21.15.188:0/686122469 learned_addr learned my addr 172.21.15.188:0/686122469 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:57.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb08488e700 1 -- 172.21.15.188:0/686122469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074005040 con 0x7fb0800f6510 2023-11-15T04:17:57.022 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb08488e700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb074013510 tx=0x7fb07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:57.023 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb074009320 con 0x7fb0800f6510 2023-11-15T04:17:57.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb080106e20 con 0x7fb0800f6510 2023-11-15T04:17:57.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb074013f00 con 0x7fb0800f6510 2023-11-15T04:17:57.024 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb074014d90 con 0x7fb0800f6510 2023-11-15T04:17:57.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.020+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0801072b0 con 0x7fb0800f6510 2023-11-15T04:17:57.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0740095f0 con 0x7fb0800f6510 2023-11-15T04:17:57.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb080045ad0 con 0x7fb0800f6510 2023-11-15T04:17:57.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb07d7fa700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:57.025 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0740410a0 con 0x7fb0800f6510 2023-11-15T04:17:57.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb07ffff700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:57.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.021+0000 7fb07ffff700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 0x7fb070032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb06c0096a0 tx=0x7fb06c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:57.026 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.026+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb07401d7f0 con 0x7fb0800f6510 2023-11-15T04:17:57.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.229+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0800fbf60 con 0x7fb0800f6510 2023-11-15T04:17:57.229 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.230+0000 7fb07d7fa700 1 -- 172.21.15.188:0/686122469 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb074010230 con 0x7fb0800f6510 2023-11-15T04:17:57.230 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:17:57.230 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:17:57.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 msgr2=0x7fb070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:57.231 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 0x7fb070032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb06c0096a0 tx=0x7fb06c006b40).stop 2023-11-15T04:17:57.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 msgr2=0x7fb080106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:57.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb074013510 tx=0x7fb07400b040).stop 2023-11-15T04:17:57.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 shutdown_connections 2023-11-15T04:17:57.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fb07002ff60 0x7fb070032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:57.232 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 --2- 172.21.15.188:0/686122469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0800f6510 0x7fb080106730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:57.233 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 >> 172.21.15.188:0/686122469 conn(0x7fb0800f1c70 msgr2=0x7fb0800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:57.233 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.231+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 shutdown_connections 2023-11-15T04:17:57.233 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:57.232+0000 7fb086af2700 1 -- 172.21.15.188:0/686122469 wait complete. 2023-11-15T04:17:57.233 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:17:57.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:57 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:56.136181+0000 mgr.smithi038.bnbogz (mgr.14164) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:58.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:58 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:17:57.230296+0000 mon.smithi038 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.188:0/686122469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:17:58.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:17:58.616 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:17:59.593 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:17:59 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:17:58.136525+0000 mgr.smithi038.bnbogz (mgr.14164) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:17:59.903 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- 172.21.15.188:0/159378043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 msgr2=0x7f2de40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:17:59.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 --2- 172.21.15.188:0/159378043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de40f55c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2dd4009fb0 tx=0x7f2dd4009b30).stop 2023-11-15T04:17:59.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- 172.21.15.188:0/159378043 shutdown_connections 2023-11-15T04:17:59.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 --2- 172.21.15.188:0/159378043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de40f55c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:17:59.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- 172.21.15.188:0/159378043 >> 172.21.15.188:0/159378043 conn(0x7f2de40f09a0 msgr2=0x7f2de40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:17:59.904 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- 172.21.15.188:0/159378043 shutdown_connections 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- 172.21.15.188:0/159378043 wait complete. 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 Processor -- start 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- start start 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.903+0000 7f2deb01b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4060240 con 0x7f2de40f51a0 2023-11-15T04:17:59.905 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2dea019700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:59.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2dea019700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39800/0 (socket says 172.21.15.188:39800) 2023-11-15T04:17:59.906 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2dea019700 1 -- 172.21.15.188:0/2095462233 learned_addr learned my addr 172.21.15.188:0/2095462233 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:17:59.907 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2dea019700 1 -- 172.21.15.188:0/2095462233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4005040 con 0x7f2de40f51a0 2023-11-15T04:17:59.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2dea019700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2dd4013510 tx=0x7f2dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:59.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dd4009320 con 0x7f2de40f51a0 2023-11-15T04:17:59.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de4060440 con 0x7f2de40f51a0 2023-11-15T04:17:59.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dd4013f00 con 0x7f2de40f51a0 2023-11-15T04:17:59.908 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dd4014dc0 con 0x7f2de40f51a0 2023-11-15T04:17:59.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.904+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de4109860 con 0x7f2de40f51a0 2023-11-15T04:17:59.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.905+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de4045ad0 con 0x7f2de40f51a0 2023-11-15T04:17:59.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.906+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dd40095f0 con 0x7f2de40f51a0 2023-11-15T04:17:59.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.906+0000 7f2ddaffd700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 0x7f2dd0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:17:59.909 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.906+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dd4041190 con 0x7f2de40f51a0 2023-11-15T04:17:59.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.906+0000 7f2de9818700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 0x7f2dd0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:17:59.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.906+0000 7f2de9818700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 0x7f2dd0032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f2de00096a0 tx=0x7f2de0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:17:59.910 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:17:59.910+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dd4014420 con 0x7f2de40f51a0 2023-11-15T04:18:00.115 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.115+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2de410a070 con 0x7f2de40f51a0 2023-11-15T04:18:00.116 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.116+0000 7f2ddaffd700 1 -- 172.21.15.188:0/2095462233 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dd401c930 con 0x7f2de40f51a0 2023-11-15T04:18:00.116 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:18:00.117 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:18:00.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 msgr2=0x7f2dd0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:00.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 0x7f2dd0032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f2de00096a0 tx=0x7f2de0006b40).stop 2023-11-15T04:18:00.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 msgr2=0x7f2de41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:00.118 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2dd4013510 tx=0x7f2dd400b040).stop 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 shutdown_connections 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f2dd002ffb0 0x7f2dd0032470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 --2- 172.21.15.188:0/2095462233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de40f51a0 0x7f2de41091b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 >> 172.21.15.188:0/2095462233 conn(0x7f2de40f09a0 msgr2=0x7f2de40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 shutdown_connections 2023-11-15T04:18:00.119 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:00.118+0000 7f2deb01b700 1 -- 172.21.15.188:0/2095462233 wait complete. 2023-11-15T04:18:00.120 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:18:00.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:00 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:18:00.116590+0000 mon.smithi038 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.188:0/2095462233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:18:01.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:18:01.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:18:01.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:01 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:18:00.136812+0000 mgr.smithi038.bnbogz (mgr.14164) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:18:02.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- 172.21.15.188:0/375215798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 msgr2=0x7fc77c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:02.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 --2- 172.21.15.188:0/375215798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fc770009fb0 tx=0x7fc770009b30).stop 2023-11-15T04:18:02.723 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- 172.21.15.188:0/375215798 shutdown_connections 2023-11-15T04:18:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 --2- 172.21.15.188:0/375215798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- 172.21.15.188:0/375215798 >> 172.21.15.188:0/375215798 conn(0x7fc77c0f0b70 msgr2=0x7fc77c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:18:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- 172.21.15.188:0/375215798 shutdown_connections 2023-11-15T04:18:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- 172.21.15.188:0/375215798 wait complete. 2023-11-15T04:18:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 Processor -- start 2023-11-15T04:18:02.725 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- start start 2023-11-15T04:18:02.725 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:18:02.725 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.722+0000 7fc78439d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c060240 con 0x7fc77c0f5410 2023-11-15T04:18:02.727 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc782139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:18:02.727 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc782139700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39816/0 (socket says 172.21.15.188:39816) 2023-11-15T04:18:02.727 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc782139700 1 -- 172.21.15.188:0/777350416 learned_addr learned my addr 172.21.15.188:0/777350416 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:18:02.728 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc782139700 1 -- 172.21.15.188:0/777350416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc770005040 con 0x7fc77c0f5410 2023-11-15T04:18:02.728 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc782139700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fc770013510 tx=0x7fc77000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:18:02.729 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc770009320 con 0x7fc77c0f5410 2023-11-15T04:18:02.729 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77c060440 con 0x7fc77c0f5410 2023-11-15T04:18:02.729 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc770013f00 con 0x7fc77c0f5410 2023-11-15T04:18:02.729 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc770014d90 con 0x7fc77c0f5410 2023-11-15T04:18:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.723+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77c109ae0 con 0x7fc77c0f5410 2023-11-15T04:18:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.724+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7700095f0 con 0x7fc77c0f5410 2023-11-15T04:18:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.724+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77c045ad0 con 0x7fc77c0f5410 2023-11-15T04:18:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.724+0000 7fc76effd700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 0x7fc768032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:18:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.724+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc770041000 con 0x7fc77c0f5410 2023-11-15T04:18:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.725+0000 7fc781938700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 0x7fc768032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:18:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.725+0000 7fc781938700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 0x7fc768032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc7780096a0 tx=0x7fc778006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:18:02.732 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.729+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc770014420 con 0x7fc77c0f5410 2023-11-15T04:18:02.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:02 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:18:02.137086+0000 mgr.smithi038.bnbogz (mgr.14164) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:18:02.936 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.936+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc77c0fae60 con 0x7fc77c0f5410 2023-11-15T04:18:02.937 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.936+0000 7fc76effd700 1 -- 172.21.15.188:0/777350416 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc770010230 con 0x7fc77c0f5410 2023-11-15T04:18:02.938 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:18:02.939 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:18:02.939 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 msgr2=0x7fc768032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:02.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 0x7fc768032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc7780096a0 tx=0x7fc778006b40).stop 2023-11-15T04:18:02.940 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 msgr2=0x7fc77c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fc770013510 tx=0x7fc77000b040).stop 2023-11-15T04:18:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 shutdown_connections 2023-11-15T04:18:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7fc76802ff60 0x7fc768032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 --2- 172.21.15.188:0/777350416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc77c0f5410 0x7fc77c109430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:02.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 >> 172.21.15.188:0/777350416 conn(0x7fc77c0f0b70 msgr2=0x7fc77c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:18:02.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 shutdown_connections 2023-11-15T04:18:02.942 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:02.938+0000 7fc78439d700 1 -- 172.21.15.188:0/777350416 wait complete. 2023-11-15T04:18:02.943 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:18:03.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:03 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:18:02.937714+0000 mon.smithi038 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.188:0/777350416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:18:04.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:18:04.366 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:18:04.842 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:04 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: cluster 2023-11-15T04:18:04.137359+0000 mgr.smithi038.bnbogz (mgr.14164) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:18:05.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 -- 172.21.15.188:0/1657190700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 msgr2=0x7f89780f4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:05.579 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 --2- 172.21.15.188:0/1657190700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89780f4cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f8970009fb0 tx=0x7f8970009b30).stop 2023-11-15T04:18:05.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 -- 172.21.15.188:0/1657190700 shutdown_connections 2023-11-15T04:18:05.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 --2- 172.21.15.188:0/1657190700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89780f4cc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:05.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 -- 172.21.15.188:0/1657190700 >> 172.21.15.188:0/1657190700 conn(0x7f89780f02e0 msgr2=0x7f89780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:18:05.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 -- 172.21.15.188:0/1657190700 shutdown_connections 2023-11-15T04:18:05.580 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.577+0000 7f897fd2c700 1 -- 172.21.15.188:0/1657190700 wait complete. 2023-11-15T04:18:05.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897fd2c700 1 Processor -- start 2023-11-15T04:18:05.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897fd2c700 1 -- start start 2023-11-15T04:18:05.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897fd2c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:18:05.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897fd2c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978106bb0 con 0x7f89780f48a0 2023-11-15T04:18:05.581 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897dac8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:18:05.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897dac8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.188:39842/0 (socket says 172.21.15.188:39842) 2023-11-15T04:18:05.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.578+0000 7f897dac8700 1 -- 172.21.15.188:0/549883441 learned_addr learned my addr 172.21.15.188:0/549883441 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-11-15T04:18:05.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f897dac8700 1 -- 172.21.15.188:0/549883441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970005040 con 0x7f89780f48a0 2023-11-15T04:18:05.582 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f897dac8700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f8970013510 tx=0x7f897000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:18:05.583 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8970009320 con 0x7f89780f48a0 2023-11-15T04:18:05.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8978106db0 con 0x7f89780f48a0 2023-11-15T04:18:05.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8970013f00 con 0x7f89780f48a0 2023-11-15T04:18:05.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8970014d90 con 0x7f89780f48a0 2023-11-15T04:18:05.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.579+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8978107240 con 0x7f89780f48a0 2023-11-15T04:18:05.584 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89700095f0 con 0x7f89780f48a0 2023-11-15T04:18:05.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978045ad0 con 0x7f89780f48a0 2023-11-15T04:18:05.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f89767fc700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:18:05.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89700410c0 con 0x7f89780f48a0 2023-11-15T04:18:05.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f897d2c7700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:18:05.585 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.580+0000 7f897d2c7700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 0x7f8960032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f896c0096a0 tx=0x7f896c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:18:05.586 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.585+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8970014420 con 0x7f89780f48a0 2023-11-15T04:18:05.788 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.788+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89781080f0 con 0x7f89780f48a0 2023-11-15T04:18:05.789 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.788+0000 7f89767fc700 1 -- 172.21.15.188:0/549883441 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8970010230 con 0x7f89780f48a0 2023-11-15T04:18:05.789 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-11-15T04:18:05.790 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"7bd2f4c8-836c-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:07:50.797737Z","created":"2023-11-15T04:07:50.797737Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi038","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:18:05.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 msgr2=0x7f8960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:05.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 0x7f8960032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f896c0096a0 tx=0x7f896c006b40).stop 2023-11-15T04:18:05.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 msgr2=0x7f89781066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:18:05.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f8970013510 tx=0x7f897000b040).stop 2023-11-15T04:18:05.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 shutdown_connections 2023-11-15T04:18:05.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:6800/3073786548,v1:172.21.15.38:6801/3073786548] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:05.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 --2- 172.21.15.188:0/549883441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89780f48a0 0x7f89781066c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:18:05.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 >> 172.21.15.188:0/549883441 conn(0x7f89780f02e0 msgr2=0x7f89780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:18:05.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 shutdown_connections 2023-11-15T04:18:05.793 INFO:teuthology.orchestra.run.smithi188.stderr:2023-11-15T04:18:05.790+0000 7f897fd2c700 1 -- 172.21.15.188:0/549883441 wait complete. 2023-11-15T04:18:05.793 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-11-15T04:18:06.092 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:05 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: audit 2023-11-15T04:18:05.789797+0000 mon.smithi038 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.188:0/549883441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:18:06.198 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T04:18:06.199 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T04:18:06.199 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:18:06.224 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:18:06.243 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T04:18:06.244 INFO:tasks.cephadm.mon.smithi038:Stopping mon.smithi038... 2023-11-15T04:18:06.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038 2023-11-15T04:18:06.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 systemd[1]: Stopping Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7... 2023-11-15T04:18:06.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:18:06.386+0000 7f9cbd7fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi038 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T04:18:06.592 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038[112367]: debug 2023-11-15T04:18:06.386+0000 7f9cbd7fa700 -1 mon.smithi038@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T04:18:06.963 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 podman[117906]: ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7-mon.smithi038 2023-11-15T04:18:06.964 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 systemd[1]: ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service: Succeeded. 2023-11-15T04:18:06.964 INFO:journalctl@ceph.mon.smithi038.smithi038.stdout:Nov 15 04:18:06 smithi038 systemd[1]: Stopped Ceph mon.smithi038 for 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7. 2023-11-15T04:18:06.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi038.service' 2023-11-15T04:18:07.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T04:18:07.052 INFO:tasks.cephadm.mon.smithi038:Stopped mon.smithi038 2023-11-15T04:18:07.052 INFO:tasks.cephadm.mon.smithi188:Stopping mon.smithi188... 2023-11-15T04:18:07.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188 2023-11-15T04:18:07.089 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188.service: Unit ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188.service not loaded. 2023-11-15T04:18:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T04:18:07.091 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi188" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 5: 'sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188' 2023-11-15T04:18:07.092 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T04:18:07.094 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/ubuntu@smithi038.front.sepia.ceph.com/crash 2023-11-15T04:18:07.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T04:18:07.142 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/ubuntu@smithi188.front.sepia.ceph.com/crash 2023-11-15T04:18:07.144 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T04:18:07.190 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T04:18:07.190 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2023-11-15T04:18:07.191 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T04:18:07.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T04:18:07.230 INFO:teuthology.orchestra.run.smithi038.stderr:grep: /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T04:18:07.233 INFO:tasks.cephadm:Compressing logs... 2023-11-15T04:18:07.233 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T04:18:07.275 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T04:18:07.305 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T04:18:07.305 INFO:teuthology.orchestra.run.smithi188.stderr:: No such file or directory 2023-11-15T04:18:07.307 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cephadm.log: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T04:18:07.308 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-15T04:18:07.308 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T04:18:07.309 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T04:18:07.311 INFO:teuthology.orchestra.run.smithi188.stderr: 2023-11-15T04:18:07.311 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m0.014s 2023-11-15T04:18:07.311 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m0.011s 2023-11-15T04:18:07.311 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.012s 2023-11-15T04:18:07.312 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi038.log 2023-11-15T04:18:07.312 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi038.bnbogz.log 2023-11-15T04:18:07.312 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi038.log: 88.3% -- replaced with /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T04:18:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi038.bnbogz.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T04:18:07.393 INFO:teuthology.orchestra.run.smithi038.stderr: 92.7% -- replaced with /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi038.bnbogz.log.gz 2023-11-15T04:18:07.468 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/7bd2f4c8-836c-11ee-8dbe-212e2dc638e7/ceph-mon.smithi038.log.gz 2023-11-15T04:18:07.469 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-15T04:18:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m0.174s 2023-11-15T04:18:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m0.251s 2023-11-15T04:18:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.024s 2023-11-15T04:18:07.470 INFO:tasks.cephadm:Archiving logs... 2023-11-15T04:18:07.471 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/ubuntu@smithi038.front.sepia.ceph.com/log 2023-11-15T04:18:07.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T04:18:07.600 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/ubuntu@smithi188.front.sepia.ceph.com/log 2023-11-15T04:18:07.605 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T04:18:07.640 INFO:tasks.cephadm:Removing cluster... 2023-11-15T04:18:07.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 --force 2023-11-15T04:19:19.911 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bd2f4c8-836c-11ee-8dbe-212e2dc638e7 --force 2023-11-15T04:19:20.227 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T04:19:20.228 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T04:19:20.252 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T04:19:20.270 INFO:tasks.cephadm:Teardown complete 2023-11-15T04:19:20.271 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 5: 'sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188' 2023-11-15T04:19:20.363 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=87d107252e3f429d9c408985342f3a0e Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 5: 'sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188' 2023-11-15T04:19:20.370 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T04:19:20.387 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T04:19:20.430 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T04:19:20.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T04:19:20.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T04:19:20.523 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T04:19:20.533 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T04:19:20.534 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:19:20.537 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:19:20.554 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-11-15T04:19:20.556 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2023-11-15T04:19:20.559 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:19:20.559 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-11-15T04:19:20.559 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 -832us[ -832us] +/- 90ms 2023-11-15T04:19:20.560 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 +93us[ +90us] +/- 68ms 2023-11-15T04:19:20.560 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv03.front.sepia.ceph.com 3 6 377 38 +612us[ +610us] +/- 77ms 2023-11-15T04:19:20.560 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:19:20.562 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:19:20.562 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2023-11-15T04:19:20.563 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -813us[ -813us] +/- 90ms 2023-11-15T04:19:20.563 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 +83us[ +83us] +/- 68ms 2023-11-15T04:19:20.563 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv03.front.sepia.ceph.com 3 6 377 37 +607us[ +611us] +/- 77ms 2023-11-15T04:19:20.563 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:19:20.564 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T04:19:20.574 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T04:19:20.574 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T04:19:20.618 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T04:19:20.725 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T04:19:20.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:19:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:19:20.856 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:19:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:19:20.888 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T04:19:20.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T04:19:20.936 INFO:teuthology.task.internal:Duration was 1451.745677 seconds 2023-11-15T04:19:20.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T04:19:20.985 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T04:19:20.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T04:19:20.989 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T04:19:21.035 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:19:21.046 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:19:21.357 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T04:19:21.357 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2023-11-15T04:19:21.358 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T04:19:21.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2023-11-15T04:19:21.388 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T04:19:21.411 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T04:19:21.412 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T04:19:21.431 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T04:19:21.496 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T04:19:21.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T04:19:21.500 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T04:19:21.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T04:19:21.599 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T04:19:21.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T04:19:21.631 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T04:19:21.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T04:19:21.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:19:21.676 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:19:21.708 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2023-11-15T04:19:21.725 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2023-11-15T04:19:21.751 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:19:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:19:21.797 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T04:19:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:19:21.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T04:19:21.826 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T04:19:21.827 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/smithi038 2023-11-15T04:19:21.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T04:19:21.888 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314/remote/smithi188 2023-11-15T04:19:21.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T04:19:21.928 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T04:19:21.929 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T04:19:21.931 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T04:19:21.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T04:19:21.991 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T04:19:21.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T04:19:22.019 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T04:19:22.019 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T04:19:22.022 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T04:19:22.039 INFO:teuthology.orchestra.run.smithi188.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 04:19 /home/ubuntu/cephtest 2023-11-15T04:19:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: 525703 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 04:19 /home/ubuntu/cephtest 2023-11-15T04:19:22.044 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T04:19:22.069 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T04:19:22.179 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T04:19:22.208 DEBUG:teuthology.nuke:shortname: smithi038 2023-11-15T04:19:22.208 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:19:22.231 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 03:46:39.261530', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhiSHYR7Wh5NTjX7HrNo6Qc/9bNi9zP2X3IjyHBlNpavBG4Y0oDlHIAuk3Re+vufcV58zJ/wt1088Kio9w+63o='} 2023-11-15T04:19:22.240 DEBUG:teuthology.nuke:shortname: smithi188 2023-11-15T04:19:22.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:19:22.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 03:46:39.260567', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBUOLvmVbHExAjWaJuVTri3zapXa5LX9p9N2JCvbVN49oMFYf7g6zXL61+zGzCR2ad0TJSm2KshAEB21zRZ1hw='} 2023-11-15T04:19:22.305 INFO:teuthology.orchestra.console.smithi038:Power off 2023-11-15T04:19:22.307 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T04:19:22.334 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2023-11-15T04:19:22.335 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:22.352 INFO:teuthology.orchestra.console.smithi188:Power off 2023-11-15T04:19:22.352 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T04:19:22.362 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2023-11-15T04:19:22.377 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2023-11-15T04:19:22.377 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:22.402 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2023-11-15T04:19:26.364 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:26.404 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:26.476 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2023-11-15T04:19:26.516 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2023-11-15T04:19:30.478 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:30.518 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T04:19:30.588 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2023-11-15T04:19:30.633 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2023-11-15T04:19:30.689 INFO:teuthology.orchestra.console.smithi038:Power off completed 2023-11-15T04:19:30.735 INFO:teuthology.orchestra.console.smithi188:Power off completed 2023-11-15T04:19:30.842 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1451.7456772327423 failure_reason: 'Command failed on smithi188 with status 5: ''sudo systemctl stop ceph-7bd2f4c8-836c-11ee-8dbe-212e2dc638e7@mon.smithi188''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=87d107252e3f429d9c408985342f3a0e status: fail success: false 2023-11-15T04:19:30.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T04:19:31.024 INFO:teuthology.run:FAIL